This section will focus on how Dubbo uses Netty to achieve network communication. From the official website, we know that Dubbo protocol uses a single long connection for network transmission, that is to say, the service caller establishes a connection with the service provider for a long time, and all the service call information passes through. When a TCP connection is used for transmission, the following problems shall be considered at least at the network layer:

1. Server, client network communication model (thread model)

2. Transmission (codec, serialization).

3. Server forwarding strategy, etc.

The network start-up process of Dubbo server is shown in the previous part. In this section, we introduce two protagonists: NettyServer and NettyClient.

Dubbo uses SPI mechanism. According to the configuration, it can support the following frameworks to implement the network communication model, such as netty3,netty4, mina and grizzly. This paper focuses on the implementation based on Netty4, and the package path is dubbo-remoting-netty4. From the above flow chart, the responsibility of NettyTransport is to call the construction method of new NettyServer to build the NettyServer object. Before going deep into the construction process of NettyServer object, first look at the class inheritance level of NettyServer:

NettyServer constructor:

public NettyServer(URL url, ChannelHandler handler) throws RemotingException {  // @1
        super(url, ChannelHandlers.wrap(handler, ExecutorUtil.setThreadName(url, SERVER_THREAD_POOL_NAME)));    // @2
}

Code @ 1: URL url: service provider URL; ChannelHandler network event handler,

That is, when the corresponding network event is triggered, the event handler is executed.

• void connected(Channel channel) throws RemotingException Connection event: when receiving the connection event from the client, the method is executed to handle the relevant business operations.
• void disconnected(Channel channel) throws RemotingException: disconnect event
• void sent(Channel channel, Object message) throws RemotingException When a writable event is triggered, the server returns the response data to the client through this method.
• void received(Channel channel, Object message) throws RemotingException When the read event is triggered, the method is executed. When the server receives the request data from the client, the method is called to perform unpacking and other operations.
• void caught(Channel channel, Throwable exception) throws RemotingException This method is called when an exception occurs.

Code @2: call ChannelHandlers.wrap to wrap the native Handler, then call the construction method of its parent class. First, set the name of the thread in the Dubbo server thread pool, and specify the prefix of the thread in the thread pool by parameter threadname. The default is: DubboServerHandler +dubbo server IP and interface number. What I'm curious about is why the channel handler needs to be packaged here? What logic has been added? With the question of the bearer, this section focuses on the content of this section: event distribution mechanism. Event dispatch mechanism refers to how to execute network events (connection, read, write) after they are triggered, whether they are executed by IO thread or dispatched to thread pool. Dubbo defines the following five event distribution mechanisms:

This paper will analyze the principle of the distribution of various events in detail. ChannelHandlers#wrapInternal

protected ChannelHandler wrapInternal(ChannelHandler handler, URL url) {
        return new MultiMessageHandler(new HeartbeatHandler(ExtensionLoader.getExtensionLoader(Dispatcher.class)
                .getAdaptiveExtension().dispatch(handler, url)));
}

Here is the typical decoration mode, MultiMessageHandler, multimessage processing Handler, heartbeat Handler, heartbeat Handler. Its main function is to process heartbeat return and heartbeat request, which is directly executed in IO thread. Each time information is received, the read event stamp of channel is updated, and the write event stamp of channel is recorded each time data is sent. The key point here is to use SPI self adaptation to return to the appropriate event dispatch mechanism. The class hierarchy of Dispatcher is shown as follows:

1. Source code analysis principle of AllDispatcher

Thread dispatch mechanism: all messages are dispatched to the thread pool, including request, response, connection event, disconnection event, heartbeat, etc.

public class AllDispatcher implements Dispatcher {
    public static final String NAME = "all";
    @Override
    public ChannelHandler dispatch(ChannelHandler handler, URL url) {
        return new AllChannelHandler(handler, url);
    }
}

It can be seen from this that the event dispatch class inheritance diagram is divided into two dimensions: Dispatcher and its corresponding ChannelHandler, such as AllChannelHandler.

1.1 WrappedChannelHandler

Next, we analyze the event dispatch mechanism, focusing on the implementation system of ChannelHandler class.

Looking at Dubbo The design of channelhanler system is a classic decorator like mode. The main problem that the above dispatcher solves is whether the related network events (connection, read (request), write (response), heartbeat request, heartbeat response) are in IO thread or in additional defined thread pool. Therefore, the main responsibility of WrappedChannelHandler is to define the logic related to thread pool, specifically in IO line If it is executed on the process or in the defined thread pool, it is customized by subclass. WrappedChannelHandler implements all methods of ChannelHandler by default. The implementation of each method directly calls the decorated Handler's methods, as shown in the following figure:

Next, focus on the implementation of member variables and construction methods of WrappedChannelHandler.

protected static final ExecutorService SHARED_EXECUTOR = Executors.newCachedThreadPool(new NamedThreadFactory("DubboSharedHandler", true));
protected final ExecutorService executor;
protected final ChannelHandler handler;
protected final URL url;

• Executorservice shared? Executor: shared thread pool, default thread pool, if If ExecutorService executor is empty, shared? Executor is used
• ExecutorService executor defined thread pool
• ChannelHandler: decorated ChannelHandler
• URL url service provider URL Let's focus on its constructor:

public WrappedChannelHandler(ChannelHandler handler, URL url) {
        this.handler = handler;
        this.url = url;
        executor = (ExecutorService) ExtensionLoader.getExtensionLoader(ThreadPool.class).getAdaptiveExtension().getExecutor(url);    // @1

        String componentKey = Constants.EXECUTOR_SERVICE_COMPONENT_KEY;
        if (Constants.CONSUMER_SIDE.equalsIgnoreCase(url.getParameter(Constants.SIDE_KEY))) {
            componentKey = Constants.CONSUMER_SIDE;
        }
        DataStore dataStore = ExtensionLoader.getExtensionLoader(DataStore.class).getDefaultExtension();
        dataStore.put(componentKey, Integer.toString(url.getPort()), executor);  // @2
    }

Code @ 1: build a thread pool. Based on the SPI mechanism, users can choose cached, eager, fixed and limited. It will be described in detail in the next section. You only need to know that a thread pool is built here. Code @ 2: cache both the server and the thread pool. In the server, the cache level of the thread pool is the service provider protocol (port): thread pool.

1.2 AllChannelHandler

Event dispatch mechanism: all network events are executed in the online process pool. The implementation mechanism must be to rewrite all network event methods of ChannelHandler and call the modified ChannelHandler to execute in the online process pool. Since AllChannelHandler is the first event dispatch mechanism, its implementation is described in detail.

1.2.1 AllChannelHandler#connected

public void connected(Channel channel) throws RemotingException {
        ExecutorService cexecutor = getExecutorService();
        try {
            cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.CONNECTED));
        } catch (Throwable t) {
            throw new ExecutionException("connect event", channel, getClass() + " error when process connected event .", t);
        }
    }

The main implementation of the connection event is to first obtain the execution thread pool. The acquisition logic is as follows: if executor = (ExecutorService) ExtensionLoader.getExtensionLoader(ThreadPool.class) Getadapteextension(). Getexecutor (URL); if the thread pool cannot be obtained, the shared thread pool will be used. As you can see, the business call of the connection event is executed asynchronously, based on the thread pool. Note: when to call, the method will be called after the server receives the client connection.

2.2.2 AllChannelHandler#disconnected

public void disconnected(Channel channel) throws RemotingException {
        ExecutorService cexecutor = getExecutorService();
        try {
            cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.DISCONNECTED));
        } catch (Throwable t) {
            throw new ExecutionException("disconnect event", channel, getClass() + " error when process disconnected event .", t);
        }
    }

Its basic implementation is the same as connected, which is to execute the business extension method corresponding to the specific disconnected event in the process pool. Note: the method will be called when the server receives the disconnection from the client.

2.2.3 AllChannelHandler#received

public void received(Channel channel, Object message) throws RemotingException {
        ExecutorService cexecutor = getExecutorService();
        try {
            cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
        } catch (Throwable t) {
            //TODO A temporary solution to the problem that the exception information can not be sent to the opposite end after the thread pool is full. Need a refactoring
            //fix The thread pool is full, refuses to call, does not return, and causes the consumer to wait for time out
        	if(message instanceof Request && t instanceof RejectedExecutionException){
        		Request request = (Request)message;
        		if(request.isTwoWay()){
        			String msg = "Server side(" + url.getIp() + "," + url.getPort() + ") threadpool is exhausted ,detail msg:" + t.getMessage();
        			Response response = new Response(request.getId(), request.getVersion());
        			response.setStatus(Response.SERVER_THREADPOOL_EXHAUSTED_ERROR);
        			response.setErrorMessage(msg);
        			channel.send(response);
        			return;
        		}
        	}
            throw new ExecutionException(message, channel, getClass() + " error when process received event .", t);
        }
    }

The timing of the call: when the server receives the request sent by the client, the IO thread (Netty) first decodes a request from the binary stream. The parameter Object message is called the request, and then is executed after the thread pool is submitted to the thread pool. After the operation is completed, the channel will be called after the result is assembled, Channel#write. Flush) method to write the response result to the channel. Note: for all event dispatch mechanism, channelhandler "receive" is executed in the online process pool.

2.2.4 AllChannelHandler#caught

public void caught(Channel channel, Throwable exception) throws RemotingException {
        ExecutorService cexecutor = getExecutorService();
        try {
            cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.CAUGHT, exception));
        } catch (Throwable t) {
            throw new ExecutionException("caught event", channel, getClass() + " error when process caught event .", t);
        }
    }

When an exception occurs, channelhandler should execute in the thread pool as well. Surprisingly, AllChannelHandler does not override the sent method of WrappedChannelHandler, that is to say, the channelhandler event callback method is executed in IO thread. WrappedChannelHandler#sent

public void sent(Channel channel, Object message) throws RemotingException {
        handler.sent(channel, message);
}

This is different from the official documents.

1.3 ExecutionChannelHandler

Corresponding event dispatcher: ExecutionDispatcher, configuration value: execution. From the implementation of its source code, it is basically similar to the AllDispatcher implementation. The only difference is that if the executor thread pool is empty, the shared thread pool will not be used. At present, I can't think of any circumstances in which the thread pool will fail to initialize.

1.4 DirectDispatcher

Direct dispatch means that all events are executed in IO thread, so its implementation is very simple:

public class DirectDispatcher implements Dispatcher {
    public static final String NAME = "direct";
    @Override
    public ChannelHandler dispatch(ChannelHandler handler, URL url) {
        return handler;
    }
}

1.5 MessageOnlyDispatcher,MessageOnlyChannelHandler

Event dispatcher: only request events are executed in the process pool, and other response events, heartbeat, connection, disconnection and other events are executed on IO threads, so it only needs to rewrite the receive method:

@Override
    public void received(Channel channel, Object message) throws RemotingException {
        ExecutorService cexecutor = executor;
        if (cexecutor == null || cexecutor.isShutdown()) {
            cexecutor = SHARED_EXECUTOR;
        }
        try {
            cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
        } catch (Throwable t) {
            throw new ExecutionException(message, channel, getClass() + " error when process received event .", t);
        }
    }

1.6 ConnectionOrderedDispatcher ConnectionOrderedChannelHandler

Event dispatcher: connect and disconnect events are queued for execution, and queue length can be set through the connect.queue.capacity property, and request events and exception events are executed in the online process pool.

1.6.1 construction method

public ConnectionOrderedChannelHandler(ChannelHandler handler, URL url) {
        super(handler, url);
        String threadName = url.getParameter(Constants.THREAD_NAME_KEY, Constants.DEFAULT_THREAD_NAME);
        connectionExecutor = new ThreadPoolExecutor(1, 1,
                0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<runnable>(url.getPositiveParameter(Constants.CONNECT_QUEUE_CAPACITY, Integer.MAX_VALUE)),
                new NamedThreadFactory(threadName, true),
                new AbortPolicyWithReport(threadName, url)
        );  // FIXME There's no place to release connectionExecutor!
        queuewarninglimit = url.getParameter(Constants.CONNECT_QUEUE_WARNING_SIZE, Constants.DEFAULT_CONNECT_QUEUE_WARNING_SIZE);
    }

Focus on the connectionExecutor, which is used to execute the thread pool of connection and disconnection events. There is only one thread in the thread pool, and the queue can choose the time bound queue. Configure it through the connect.queue.capacity property. If the event exceeds, the execution will be rejected.

1.6.2 ConnectionOrderedChannelHandler#connected

public void connected(Channel channel) throws RemotingException {
        try {
            checkQueueLength();
            connectionExecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.CONNECTED));
        } catch (Throwable t) {
            throw new ExecutionException("connect event", channel, getClass() + " error when process connected event .", t);
        }
    }

Check the queue length. If the warning value is exceeded, the warning information will be output and then submitted to the connection thread pool for execution. The disconnected event is similar. Other received and caught events are the same as AllDispatcher and are not repeated.

Conclusion: This paper mainly analyzes and expounds the Dubbo Dispatch mechanism, but it is different from the official documents. First, it is summarized as follows: Dispatch: all sent event methods and heartbeat requests are executed on the IO thread.

1. all Except for send event callback method and heartbeat, all processes are executed on the process pool.
2. execution Similar to all, the only area is when the all thread pool is not specified, the shared thread pool can be used. This difference is equivalent to No.
3. message Only request events are executed in the thread pool, others are executed on IO threads.
4. connection Request events are executed in the thread pool, connection and disconnection events are queued for execution (including the thread pool of one thread)
5. direct All events are executed in the IO thread.

The author introduces: Ding Wei, the author of "RocketMQ technology insider", RocketMQ community outstanding evangelist, CSDN2019 blogger TOP10, maintains the official account: Middleware interest circle At present, source code analysis Java collection, Java Concurrent contract (JUC), Netty, Mycat, Dubbo, RocketMQ, Mybatis and other source code columns have been published successively. You can click the link to join Middleware knowledge planet , discuss high concurrent and distributed service architecture and exchange source code together. </runnable>