Problems in dubbo registering jvm to close hook

Keywords: Dubbo Java Spring jvm

dubbo performs its elegant downtime by registering the jvm to close the hook during downtime. However, when dubbo runs with spring, because spring also closes the hook registration through the jvm:

public abstract class AbstractApplicationContext:
@Override
	public void registerShutdownHook() {
		if (this.shutdownHook == null) {
			// No shutdown hook registered yet.
			this.shutdownHook = new Thread() {
				@Override
				public void run() {
					synchronized (startupShutdownMonitor) {
						doClose();
					}
				}
			};
			Runtime.getRuntime().addShutdownHook(this.shutdownHook);
		}
	}

The execution of jvm hook function is concurrent.

class Shutdown:
 /* Run all registered shutdown hooks
     */
    private static void runHooks() {
        for (int i=0; i < MAX_SYSTEM_HOOKS; i++) {
            try {
                Runnable hook;
                synchronized (lock) {
                    // acquire the lock to make sure the hook registered during
                    // shutdown is visible here.
                    currentRunningHook = i;
                    hook = hooks[i];
                }
                if (hook != null) hook.run();
            } catch(Throwable t) {
                if (t instanceof ThreadDeath) {
                    ThreadDeath td = (ThreadDeath)t;
                    throw td;
                }
            }
        }
    }

If the closing hook of spring is executed before the closing hook registered by dubbo, an IllegalStateException will be reported when dubbo closes the execution of the hook, for example:

[DubboShutdownHook] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.alibaba.dubbo.remoting.exchange.support.DefaultFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.alibaba.dubbo.remoting.exchange.support.DefaultFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1311)
	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1299)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1158)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
	at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.isRunning(HeaderExchangeServer.java:88)
	at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.close(HeaderExchangeServer.java:109)
	at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.destroy(DubboProtocol.java:384)
	at com.alibaba.dubbo.rpc.protocol.ProtocolListenerWrapper.destroy(ProtocolListenerWrapper.java:72)
	at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper.destroy(ProtocolFilterWrapper.java:105)
	at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:153)
	at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:82)
	at java.lang.Thread.run(Thread.java:748)

The above errors mainly occurred in dubbo 2.5.x. in 2.6.x, in addition to registering the close hook, the spring close event was also monitored:

SpringExtensionFactory:
private static class ShutdownHookListener implements ApplicationListener {
        @Override
        public void onApplicationEvent(ApplicationEvent event) {
            if (event instanceof ContextClosedEvent) {
                // we call it anyway since dubbo shutdown hook make sure its destroyAll() is re-entrant.
                // pls. note we should not remove dubbo shutdown hook when spring framework is present, this is because
                // its shutdown hook may not be installed.
                DubboShutdownHook shutdownHook = DubboShutdownHook.getDubboShutdownHook();
                shutdownHook.destroyAll();
            }
        }
    }

Execute the close function when listening for a container close event.
At the same time, the jvm's closing hook has not been removed, which is for compatibility with non spring container environment.

In version 2.7.4, it goes further:

public class SpringExtensionFactory:
 public static void addApplicationContext(ApplicationContext context) {
        CONTEXTS.add(context);
        if (context instanceof ConfigurableApplicationContext) {
            ((ConfigurableApplicationContext) context).registerShutdownHook();
            DubboShutdownHook.getDubboShutdownHook().unregister();
        }
        BeanFactoryUtils.addApplicationListener(context, SHUTDOWN_HOOK_LISTENER);
    }

When using the container context, if the context implements spring's ConfigurableApplicationContext, register the spring shutdown hook explicitly in this context, add dubbo ShutdownHookListener in the container, and log off the shutdown hook registered by dubbo in the jvm.
And use

private final AtomicBoolean registered = new AtomicBoolean(false);

It ensures that the registration will not be repeated even when the jvm registration hook is called after the ApplicationListener is added. It is compatible with the spring context or not.

As mentioned earlier, in the 2.5.x version, when spring closes the problems caused by the execution of hooks and dubbo hooks, it is necessary to solve the problem in this version, and call the dubbo closing function in the spring shutdown event.

@Component("dubboDestroyHandler")
public class DubboDestroyHandler implements ApplicationListener {
    private static final Logger LOGGER = LoggerFactory.getLogger(DubboDestroyHandler.class);

    @Override
    public void onApplicationEvent(ApplicationEvent event) {
        //If it is a container close event
        if (event instanceof ContextClosedEvent) {
            LOGGER.info("Explicit call dubbo destroyAll");
            ProtocolConfig.destroyAll();
        }
    }
}

However, there is a hidden danger. Before version 2.6.3, the method to determine whether the client still holds the connection is as follows:

public class HeaderExchangeServer:
 private boolean isRunning() {
        Collection<Channel> channels = getChannels();
        for (Channel channel : channels) {
            if (DefaultFuture.hasFuture(channel)) {
                return true;
            }
        }
        return false;
    }

DefaultFuture.hasFuture(channel) can't accurately judge whether the client still holds the connection, which causes the channel to close prematurely, which may lead to incomplete business. Until 2.6.3, it was changed to:

            if (channel.isConnected()) {
                return true;
            }

In the 2.5 series after 2.5.9, after the server unregisters from the registry, it waits for a while for the consumer to receive the notice of service change.

public class ProtocolConfig extends AbstractConfig:
// Wait for registry notification
        try {
            Thread.sleep(ConfigUtils.getServerShutdownTimeout());
        } catch (InterruptedException e) {
            logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
        }

In the version before 2.5.9, consumers can only be expected to receive timely notification from the registry to disable the channel. In the case of small cluster size and small burden of registration center, there is generally no problem. But to be on the safe side, upgrading to a higher version will fundamentally solve the problem

Posted by Branden Wagner on Sun, 27 Oct 2019 02:24:49 -0700