I. Introduction
In "ShutdownHook- Java elegant solutions to stop," the article we talked to the Java implementation of the principle of elegance downtime. Next we just based on the above knowledge, in-depth internal Dubbo, Dubbo to look at how to achieve elegant downtime.
Two, Dubbo elegant halting problem to be solved
In order to achieve elegant downtime, Dubbo need to solve some of the problems:
- New request can not be sent to Dubbo service providers are shut down.
- If you turn off a service provider, the service request has been received, processed needs to offline services.
- If you turn off service to consumers, the service request has been issued, waiting for the response to return.
To solve three problems above, in order to make the business impact of downtime to a minimum, so elegant stop.
Three, 2.5.X
Dubbo elegant achieved relatively complete shutdown in the 2.5.X version, this version is relatively simple, relatively easy to understand. So we first Dubbo 2.5.X versions of source code as the basis, first look at how to achieve Dubbo elegant downtime.
3.1, elegantly down the overall implementation
Class located in the inlet elegant stop AbstractConfigstatic code, the source code is as follows:
static {
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
public void run() {
if (logger.isInfoEnabled()) {
logger.info("Run shutdown hook now.");
}
ProtocolConfig.destroyAll();
}
}, "DubboShutdownHook"));
}
复制代码Here will be a registration ShutdownHook, application shutdown will be triggered once the call ProtocolConfig.destroyAll().
ProtocolConfig.destroyAll()Source as follows:
public static void destroyAll() {
// 防止并发调用
if (!destroyed.compareAndSet(false, true)) {
return;
}
// 先注销注册中心
AbstractRegistryFactory.destroyAll();
// Wait for registry notification
try {
Thread.sleep(ConfigUtils.getServerShutdownTimeout());
} catch (InterruptedException e) {
logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
}
ExtensionLoader<Protocol> loader = ExtensionLoader.getExtensionLoader(Protocol.class);
// 再注销 Protocol
for (String protocolName : loader.getLoadedExtensions()) {
try {
Protocol protocol = loader.getLoadedExtension(protocolName);
if (protocol != null) {
protocol.destroy();
}
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
复制代码Can be seen from the above, Dubbo elegant downtime is primarily divided into two steps:
- Cancellation of registration centers
- Cancellation of all
Protocol
3.2 cancellation of registration centers
Logout registry source as follows:
public static void destroyAll() {
if (LOGGER.isInfoEnabled()) {
LOGGER.info("Close all registries " + getRegistries());
}
// Lock up the registry shutdown process
LOCK.lock();
try {
for (Registry registry : getRegistries()) {
try {
registry.destroy();
} catch (Throwable e) {
LOGGER.error(e.getMessage(), e);
}
}
REGISTRIES.clear();
} finally {
// Release the lock
LOCK.unlock();
}
}
复制代码This method will be written off internally generated registry service. Cancellation of registration centers internal logic is relatively simple, there is no longer deep source, the direct use of Photo Gallery.
ps: Source is located:
AbstractRegistry
To ZK, for example, Dubbo will delete the corresponding service node, and then unsubscribe. Since the information is changed node ZK, ZK will notify the server of the service offline dubbo consumer node, and finally close the service with ZK connection.
By registering Center, Dubbo may notify consumers offline services, new requests are no longer sent down the line of nodes, would resolve the first problem mentioned above: the new request can not be sent to Dubbo being shut down service provider.
But here there are some drawbacks, due to network isolation, ZK server and Dubbo connection there may be some delay, ZK inform the consumer may not notice the first time at the end. Given this situation, after the cancellation of registration centers, waiting to join binary code is as follows:
// Wait for registry notification
try {
Thread.sleep(ConfigUtils.getServerShutdownTimeout());
} catch (InterruptedException e) {
logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
}
复制代码The default wait time is 10000ms , it can be provided dubbo.service.shutdown.waitto override the default parameters. 10s only an empirical value may be set according to actual situation. But the wait time is set relatively stress, can not be set too short, too short will cause the consumer side has not received notice ZK providers to shut down. Can not be set too long, too long will lead to shutting down the application side length of time, affecting publishing experience.
3.3, write-off Protocol
ExtensionLoader<Protocol> loader = ExtensionLoader.getExtensionLoader(Protocol.class);
for (String protocolName : loader.getLoadedExtensions()) {
try {
Protocol protocol = loader.getLoadedExtension(protocolName);
if (protocol != null) {
protocol.destroy();
}
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
复制代码loader#getLoadedExtensionsIt will return two Protocolsub-categories, namely DubboProtocolwith InjvmProtocol.
DubboProtocolIt interacts with the server requests, and InjvmProtocolan internal request interaction. If the application calls the internal method Dubbo provide services themselves, will not perform network calls, direct execution.
Here we analyze the DubboProtocolinternal logic.
DubboProtocol#destroy Source:
public void destroy() {
// 关闭 Server
for (String key : new ArrayList<String>(serverMap.keySet())) {
ExchangeServer server = serverMap.remove(key);
if (server != null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo server: " + server.getLocalAddress());
}
server.close(ConfigUtils.getServerShutdownTimeout());
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
// 关闭 Client
for (String key : new ArrayList<String>(referenceClientMap.keySet())) {
ExchangeClient client = referenceClientMap.remove(key);
if (client != null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo connect: " + client.getLocalAddress() + "-->" + client.getRemoteAddress());
}
client.close(ConfigUtils.getServerShutdownTimeout());
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
for (String key : new ArrayList<String>(ghostClientMap.keySet())) {
ExchangeClient client = ghostClientMap.remove(key);
if (client != null) {
try {
if (logger.isInfoEnabled()) {
logger.info("Close dubbo connect: " + client.getLocalAddress() + "-->" + client.getRemoteAddress());
}
client.close(ConfigUtils.getServerShutdownTimeout());
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
stubServiceMethodsMap.clear();
super.destroy();
}
复制代码Dubbo default Netty as its underlying communication framework, divided Serverwith Client. ServerOther consumers for receiving Clienta request issued.
In the above first source off Server, stop receiving a new request, and then closed Client. Doing so reduces the likelihood of the service being invoked consumers.
3.4, close the Server
First will be called HeaderExchangeServer#close, source code as follows:
public void close(final int timeout) {
startClose();
if (timeout > 0) {
final long max = (long) timeout;
final long start = System.currentTimeMillis();
if (getUrl().getParameter(Constants.CHANNEL_SEND_READONLYEVENT_KEY, true)) {
// 发送 READ_ONLY 事件
sendChannelReadOnlyEvent();
}
while (HeaderExchangeServer.this.isRunning()
&& System.currentTimeMillis() - start < max) {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
logger.warn(e.getMessage(), e);
}
}
}
// 关闭定时心跳检测
doClose();
server.close(timeout);
}
private void doClose() {
if (!closed.compareAndSet(false, true)) {
return;
}
stopHeartbeatTimer();
try {
scheduled.shutdown();
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
复制代码Here it will be sent to the service consumer READ_ONLYevents. After consumer acceptance, active exclude this node, the request is sent to the other normal nodes. This in turn further reduces the impact of the delay caused by the notification registry.
Next will receive heartbeat, the underlying communication framework close NettyServer. Here we will call the NettyServer#closemethod, which actually AbstractServerachieved at.
AbstractServer#close Source as follows:
public void close(int timeout) {
ExecutorUtil.gracefulShutdown(executor, timeout);
close();
}
复制代码Here first turn off the service thread pool, this process will be completed as task execution thread pool, and then close the thread pool, and then finally closed Netty communication underlying Server.
Dubbo will be the default request / requests dispatched to the heart and other business processing thread pool.
Close Server, elegant waiting thread pool closed to solve the second problem mentioned above: If you turn off a service provider, the service request has been received, processed needs to offline services.
Dubbo service provider shut down process shown in Figure:
ps: In order to facilitate debugging source code, attach a close call with Server.
DubboProtocol#destroy
->HeaderExchangeServer#close
->AbstractServer#close
->NettyServer#doClose
复制代码Close Client 3.5
Client Server with a substantially closed manner, this mainly introduce processing logic has issued a request, the code is located HeaderExchangeChannel#close.
// graceful close
public void close(int timeout) {
if (closed) {
return;
}
closed = true;
if (timeout > 0) {
long start = System.currentTimeMillis();
// 等待发送的请求响应信息
while (DefaultFuture.hasFuture(channel)
&& System.currentTimeMillis() - start < timeout) {
try {
Thread.sleep(10);
} catch (InterruptedException e) {
logger.warn(e.getMessage(), e);
}
}
}
close();
}
复制代码Client closed when the request for information if there is no response is received, it will wait for a certain time, until it is confirmed that all requests receive a response, or wait longer than the timeout.
ps: Dubbo request will be temporarily stored in
DefaultFuturea Map, so you can simply determine what Map know whether the request will receive a response.
By this point we have solved the third problem: If you turn off service to consumers, the service request has been issued, waiting for the response to return.
Stop Dubbo elegant overall process shown in FIG.
ps: Client Close call chain as follows:
DubboProtocol#close
->ReferenceCountExchangeClient#close
->HeaderExchangeChannel#close
->AbstractClient#close
复制代码Four, 2.7.X
Dubbo is generally used in conjunction with the Spring framework, 2.5.X versions of the shutdown process may lead to an elegant stop failure. This is because the event will trigger the corresponding ShutdownHook closed when the Spring framework, logout Bean. If this process is the first implementation of downtime Spring, logout Bean. And then close the event in Dubbo reference to the Spring Bean, which will make an exception occurs during shutdown, resulting in elegant downtime failure.
To solve this problem, Dubbo in version 2.6.X start reconstruction this part of the logic, and continues to iterate until 2.7.X versions.
The new version of the new ShutdownHookListener, inheritance Spring ApplicationListenerinterfaces to listen for Spring related events. Here ShutdownHookListeneris just off event listener Spring, Spring began when closed, it will trigger ShutdownHookListenerinternal logic.
public class SpringExtensionFactory implements ExtensionFactory {
private static final Logger logger = LoggerFactory.getLogger(SpringExtensionFactory.class);
private static final Set<ApplicationContext> CONTEXTS = new ConcurrentHashSet<ApplicationContext>();
private static final ApplicationListener SHUTDOWN_HOOK_LISTENER = new ShutdownHookListener();
public static void addApplicationContext(ApplicationContext context) {
CONTEXTS.add(context);
if (context instanceof ConfigurableApplicationContext) {
// 注册 ShutdownHook
((ConfigurableApplicationContext) context).registerShutdownHook();
// 取消 AbstractConfig 注册的 ShutdownHook 事件
DubboShutdownHook.getDubboShutdownHook().unregister();
}
BeanFactoryUtils.addApplicationListener(context, SHUTDOWN_HOOK_LISTENER);
}
// 继承 ApplicationListener，这个监听器将会监听容器关闭事件
private static class ShutdownHookListener implements ApplicationListener {
@Override
public void onApplicationEvent(ApplicationEvent event) {
if (event instanceof ContextClosedEvent) {
DubboShutdownHook shutdownHook = DubboShutdownHook.getDubboShutdownHook();
shutdownHook.doDestroy();
}
}
}
}
复制代码After the start of the Spring Framework initialization, will trigger SpringExtensionFactorylogic, after the write-off will be AbstractConfigregistered ShutdownHook, then increase ShutdownHookListener. This perfect solution "double hook" questions above.
Fifth, and finally
Shutdown looks elegant is not difficult to achieve, but there minutiae design is very much a point to realize there is a problem, it will lead to an elegant stop failure. If you're a graceful shutdown may wish to refer to Dubbo implementation logic.
Dubbo series recommended
1. If someone asked you in Dubbo registry works, put the article to him
2. Do not know how to implement dynamic discovery of services? Take a look at how Dubbo is done
3.Dubbo Zk data structure
4. The origin of Dubbo, talk about Spring XML Schema extension mechanism
Help articles
I welcome the attention of the public number: Interpreter program, and getting daily dry push. If you are interested in my topic content, you can focus on my blog: studyidea.cn