Netty -- source code implementation of master-slave Reactor multithreading mode
Overview
What is EventLoopGroup?
EventLoopGroup is a container for storing EventLoop, and it should have the function of thread pool.
Because EventLoopGroup indirectly inherits the ScheduledExecutorService interface, its implementation class should have the function of thread pool.
Take a look at the core properties of NioEventLoopGroup
// Default thread pool size
private static final int DEFAULT_EVENT_LOOP_THREADS;
static {
DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
"io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2)); // CPU cores x 2
}
// Array to store EventLoop
private final EventExecutor[] children;
Construction method
// If the incoming nThread is empty, the default thread pool size (CPU cores x 2) is used
protected MultithreadEventLoopGroup(int nThreads, ThreadFactory threadFactory, Object... args) {
super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, threadFactory, args);
}
// Final construction method
protected MultithreadEventExecutorGroup(int nThreads, Executor executor,
EventExecutorChooserFactory chooserFactory, Object... args) {
// Omit...
children = new EventExecutor[nThreads]; // Initialize the size of the EventExecutor array
for (int i = 0; i < nThreads; i ++) {
boolean success = false;
try {
children[i] = newChild(executor, args); // Initialize EventLoop
success = true;
} catch (Exception e) {
throw new IllegalStateException("failed to create a child event loop", e);
} finally {
// Omit...
}
}
// Omit...
}
After the NioEventLoopGroup instance is created, the size of its EventExecutor array and its stored EventLoop have been initialized.
What is EventLoop?
EventLoop is a sub interface of EventExecutor.
Take a look at the core properties of NioEventLoop
private Selector selector;
private volatile Thread thread;
private final Executor executor;
private final Queue<Runnable> taskQueue;
static final int DEFAULT_MAX_PENDING_EXECUTOR_TASKS = Math.max(16,
SystemPropertyUtil.getInt("io.netty.eventexecutor.maxPendingTasks", Integer.MAX_VALUE));
Every NioEventLoop has a Selector, thread, Executor, Queue property. When an EventLoop instance is created, its thread property is still NULL, and no thread has been created.
Channel
Take a look at the core properties of Channel
private volatile EventLoop eventLoop; // Reference to EventLoop
Each Channel has a reference to an EventLoop, that is, each Channel will be bound to an EventLoop, and its eventLoop() method returns the EventLoop bound to this Channel.
The core process after Netty starts the server
// Create bossGroup and workerGroup EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup();
// Bind port, start service serverBootstrap.bind(8888).sync();
doBind() method of AbstarctBootstrap
private ChannelFuture doBind(final SocketAddress localAddress) {
final ChannelFuture regFuture = initAndRegister(); // Initialize the Channel and register the Channel with the Selector in the EventLoop in the bossGroup
final Channel channel = regFuture.channel();
if (regFuture.cause() != null) {
return regFuture;
}
if (regFuture.isDone()) {
ChannelPromise promise = channel.newPromise();
doBind0(regFuture, channel, localAddress, promise);
return promise;
} else {
final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);
regFuture.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception { // When the initAndRegister() method is completed, it is automatically called by the IO thread
Throwable cause = future.cause();
if (cause != null) {
promise.setFailure(cause);
} else {
promise.registered();
doBind0(regFuture, channel, localAddress, promise); // Bind operation
}
}
});
return promise;
}
}
final ChannelFuture initAndRegister() {
Channel channel = null;
try {
channel = channelFactory.newChannel(); // Create NioServerSocketChannel
init(channel); // Initialize Channel
} catch (Throwable t) {
if (channel != null) {
channel.unsafe().closeForcibly();
return new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE).setFailure(t);
}
return new DefaultChannelPromise(new FailedChannel(), GlobalEventExecutor.INSTANCE).setFailure(t);
}
// The config() method returns AbstractBootStrap
// AbstractBootStrap stores bossGroup,ServerBootStrap stores workerGroup
ChannelFuture regFuture = config().group().register(channel);
if (regFuture.cause() != null) {
if (channel.isRegistered()) {
channel.close();
} else {
channel.unsafe().closeForcibly();
}
}
return regFuture;
}
The register() method of EventLoopGroup extracts a EventLoop from the EventLoopGroup through the selector, and then calls the register() method of EventLoop.
// register() method of EventLoopGroup
public ChannelFuture register(Channel channel) {
return next().register(channel); // Call the register() method of EventLoop
}
/**
* Take an EventLoop from the EventLoopGroup through the selector
*/
public EventExecutor next() {
return chooser.next();
}
register() method of SingleThreadEventLoop
public ChannelFuture register(final ChannelPromise promise) {
ObjectUtil.checkNotNull(promise, "promise");
promise.channel().unsafe().register(this, promise); // Call the Channel's register() method to pass the current object, that is, the current EventLoop instance
return promise;
}
register() method of abstarcchannel
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
if (eventLoop == null) {
throw new NullPointerException("eventLoop");
}
if (isRegistered()) {
promise.setFailure(new IllegalStateException("registered to an event loop already"));
return;
}
if (!isCompatible(eventLoop)) {
promise.setFailure(new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
return;
}
AbstractChannel.this.eventLoop = eventLoop; // Initialize the EventLoop property of the Channel
// On the first call, the thread in EventLoop is empty, so inEventLoop() returns false
if (eventLoop.inEventLoop()) {
register0(promise);
} else {
try {
eventLoop.execute(new Runnable() { // Call the execute() method of EventLoop, and put the registration operation as a task into the Runnable instance
@Override
public void run() {
// The register0() method will eventually call the doRegister() method to register the Channel with the Selector in the EventLoop
register0(promise);
}
});
} catch (Throwable t) {
logger.warn("Force-closing a channel whose registration task was not accepted by an event loop: {}",AbstractChannel.this, t);
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t);
}
}
}
execute() method of SingleThreadEventExecutor
public void execute(Runnable task) {
if (task == null) {
throw new NullPointerException("task");
}
boolean inEventLoop = inEventLoop();
addTask(task); // Put the task into the task queue of the current EventLoop
if (!inEventLoop) {
startThread(); // Start a thread and finally call the doStartThread() method
if (isShutdown()) {
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && wakesUpForTask(task)) {
wakeup(inEventLoop);
}
}
doStartThread() method of SingleThreadEventExecutor
private void doStartThread() {
assert thread == null; // The thread in the current EventLoop is empty, and the output result must be true
executor.execute(new Runnable() { // Submit a task to the thread pool, and then create a thread to execute asynchronously at the same time
@Override
public void run() {
thread = Thread.currentThread(); // Initializes the thread property in the EventLoop, which is the thread in the thread pool that performs the task
if (interrupted) {
thread.interrupt();
}
boolean success = false;
updateLastExecutionTime();
try {
SingleThreadEventExecutor.this.run(); // Core method (current object is NioEventLoop)
success = true;
} catch (Throwable t) {
logger.warn("Unexpected exception from an event executor: ", t);
} finally {
// Omit...
}
}
});
}
run() method of NioEventLoop
protected void run() {
for (;;) { // Dead cycle
try {
try {
switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
case SelectStrategy.CONTINUE:
continue;
case SelectStrategy.BUSY_WAIT:
case SelectStrategy.SELECT:
select(wakenUp.getAndSet(false));
if (wakenUp.get()) {
selector.wakeup();
}
default:
}
} catch (IOException e) {
rebuildSelector0();
handleLoopException(e);
continue;
}
cancelledKeys = 0;
needsToSelectAgain = false;
final int ioRatio = this.ioRatio;
if (ioRatio == 100) {
try {
processSelectedKeys(); // Handle the ready event of the Channel, and finally call the processSelectedKeysOptimized() method
} finally {
runAllTasks(); // Perform all tasks saved in the task queue
}
} else {
final long ioStartTime = System.nanoTime();
try {
processSelectedKeys();
} finally {
// Ensure we always run tasks.
final long ioTime = System.nanoTime() - ioStartTime;
runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
}
}
} catch (Throwable t) {
handleLoopException(t);
}
try {
if (isShuttingDown()) {
closeAll();
if (confirmShutdown()) {
return;
}
}
} catch (Throwable t) {
handleLoopException(t);
}
}
}
private void processSelectedKeysOptimized() {
for (int i = 0; i < selectedKeys.size; ++i) { // Traverse the SelectionKey collection corresponding to the ready Channel. If the Channel has no event ready, the collection is empty
final SelectionKey k = selectedKeys.keys[i];
selectedKeys.keys[i] = null;
final Object a = k.attachment();
if (a instanceof AbstractNioChannel) {
processSelectedKey(k, (AbstractNioChannel) a);
} else {
@SuppressWarnings("unchecked")
NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
processSelectedKey(k, task);
}
if (needsToSelectAgain) {
selectedKeys.reset(i + 1);
selectAgain();
i = -1;
}
}
}
processSelectedKey() method of NioEventLoop
private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
if (!k.isValid()) {
final EventLoop eventLoop;
try {
eventLoop = ch.eventLoop();
} catch (Throwable ignored) {
return;
}
if (eventLoop != this || eventLoop == null) {
return;
}
unsafe.close(unsafe.voidPromise());
return;
}
try {
int readyOps = k.readyOps(); // Ready events
if ((readyOps & SelectionKey.OP_CONNECT) != 0) { // Connection ready
int ops = k.interestOps();
ops &= ~SelectionKey.OP_CONNECT;
k.interestOps(ops);
unsafe.finishConnect(); // Ready to process connection
}
if ((readyOps & SelectionKey.OP_WRITE) != 0) { // Write in order
ch.unsafe().forceFlush(); // Handle write ready
}
if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) { // Receive and read ready
unsafe.read(); // Process receive and read ready
}
} catch (CancelledKeyException ignored) {
unsafe.close(unsafe.voidPromise());
}
}
When the thread in the EventLoop processes the ready event of the Channel, it will execute all the tasks saved in the task queue, and the registration task will be executed
private void register0(ChannelPromise promise) {
try {
if (!promise.setUncancellable() || !ensureOpen(promise)) {
return;
}
boolean firstRegistration = neverRegistered;
doRegister(); // Perform registration operation
neverRegistered = false;
registered = true;
pipeline.invokeHandlerAddedIfNeeded();
safeSetSuccess(promise); // ChannelFuture is set to success, and isDone() method will return true
pipeline.fireChannelRegistered();
if (isActive()) {
if (firstRegistration) {
pipeline.fireChannelActive();
} else if (config().isAutoRead()) {
beginRead();
}
}
} catch (Throwable t) {
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t); // ChannelFuture is set to fail and isDone() method returns true
}
}
doRegister() method of AbstractNioChannel
protected void doRegister() throws Exception {
boolean selected = false;
for (;;) {
try {
// Register Channel to Selector in EventLoop
selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this); // Register the Channel to the Selector, and pass the event type to be monitored to the Selector after the Channel completes the corresponding operation
return;
} catch (CancelledKeyException e) {
if (!selected) {
eventLoop().selectNow();
selected = true;
} else {
throw e;
}
}
}
}
Go back to the doBind() method of the bstarctbootstrap at the beginning
private ChannelFuture doBind(final SocketAddress localAddress) {
final ChannelFuture regFuture = initAndRegister();
final Channel channel = regFuture.channel();
if (regFuture.cause() != null) {
return regFuture;
}
if (regFuture.isDone()) { // Since the registration task is executed asynchronously, and the task has not been executed yet, the isDone() method will return false
ChannelPromise promise = channel.newPromise();
doBind0(regFuture, channel, localAddress, promise);
return promise;
} else {
final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);
regFuture.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception { // When the registration task is completed, the IO thread will automatically
Throwable cause = future.cause();
if (cause != null) {
promise.setFailure(cause);
} else {
promise.registered();
doBind0(regFuture, channel, localAddress, promise); // Bind operation
}
}
});
return promise;
}
}
private static void doBind0(
final ChannelFuture regFuture, final Channel channel,
final SocketAddress localAddress, final ChannelPromise promise) {
channel.eventLoop().execute(new Runnable() { // Similarly, get the EventLoop bound by the Channel, call the execute() method of the EventLoop, and put the binding operation into the Runnable instance as a task
@Override
public void run() {
if (regFuture.isSuccess()) {
channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);
} else {
promise.setFailure(regFuture.cause());
}
}
});
}
public void execute(Runnable task) {
if (task == null) {
throw new NullPointerException("task");
}
boolean inEventLoop = inEventLoop(); // The inEventLoop() method returns true because the current method is called by the thread that processes the registration task, that is, the thread in the EventLoop
addTask(task); // Put the task in the queue and wait for it to be executed
if (!inEventLoop) { // No execution, so far only one thread has been opened
startThread();
if (isShutdown()) {
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && wakesUpForTask(task)) {
wakeup(inEventLoop);
}
}
This is the core process after Netty starts the server
1. Create ServerSocketChannel and initialize it.
2. call the register() method of bossGroup. The method selects a EventLoop from the bossGroup through the selector, and then calls the register() method of EventLoop.
3. finally, we call the register() method of Channel, initialize the eventLoop property of Channel, then register Channel into Selector in bossGroup EventLoop as a task, put it into Runnable instance, and then call EventLoop execute (EventLoop) method.
4. The execute() method puts the task into the task queue, and then submits a task to the thread pool. At this time, a thread is created and the thread property in the EventLoop is initialized.
The 5. task initializes the thread property of EventLoop, then calls the run() method of NioEventLoop, handles the ready events of Channel and executes all tasks saved in the task queue.
6. when the registration task is executed, the thread will call back the operationComplete() method in ChannelFutureListener, bind the operation as a task, and then invoke the execute(Runnable) method of EventLoop.
7. Repeat step 4 to put the task into the task queue. Since the current thread is the thread in the EventLoop, the inEventLoop() method returns true and will not submit the task to the thread pool. The task waits to be executed by the thread in the EventLoop.
Netty's core process for handling receive and read ready events
Thread in EventLoop in BossGroup is processing Channel ready events and executing all tasks saved in task queue
// Create a connection SocketChannel socketChannel = SocketChannel.open(); socketChannel.connect(new InetSocketAddress(InetAddress.getLocalHost(), 8888));
The Selector in EventLoop in BossGroup listens for receiving ready events in ServerSocketChannel
// processSelectedKey() method of NioEventLoop
if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
unsafe.read();
}
public void read() {
assert eventLoop().inEventLoop(); // It must be true
final ChannelConfig config = config();
final ChannelPipeline pipeline = pipeline();
final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
allocHandle.reset(config);
boolean closed = false;
Throwable exception = null;
try {
try {
do {
int localRead = doReadMessages(readBuf); // Read data in Channel
if (localRead == 0) {
break;
}
if (localRead < 0) {
closed = true;
break;
}
allocHandle.incMessagesRead(localRead);
} while (allocHandle.continueReading());
} catch (Throwable t) {
exception = t;
}
int size = readBuf.size();
for (int i = 0; i < size; i ++) {
readPending = false;
pipeline.fireChannelRead(readBuf.get(i)); // Finally, the register() method of ServerBootStrap will be called
}
readBuf.clear();
allocHandle.readComplete();
pipeline.fireChannelReadComplete();
if (exception != null) {
closed = closeOnReadError(exception);
pipeline.fireExceptionCaught(exception);
}
if (closed) {
inputShutdown = true;
if (isOpen()) {
close(voidPromise());
}
}
} finally {
if (!readPending && !config.isAutoRead()) {
removeReadOp();
}
}
}
doReadMessage() method of NioServerSocketChannel
protected int doReadMessages(List<Object> buf) throws Exception {
SocketChannel ch = SocketUtils.accept(javaChannel()); // Receive the connection and call the accept() method of ServerSocketChannel
try {
if (ch != null) {
buf.add(new NioSocketChannel(this, ch)); // Put the received connection into the buffer
return 1;
}
} catch (Throwable t) {
logger.warn("Failed to create a new channel from an accepted socket.", t);
try {
ch.close();
} catch (Throwable t2) {
logger.warn("Failed to close a socket.", t2);
}
}
return 0;
}
register() method of ServerBootstrap
public void channelRead(ChannelHandlerContext ctx, Object msg) {
final Channel child = (Channel) msg;
child.pipeline().addLast(childHandler);
setChannelOptions(child, childOptions, logger);
setAttributes(child, childAttrs);
try {
// Calling the register() method of workerGroup, the method selects a EventLoop from the workerGroup through the selector, then calls the register() method of EventLoop, and finally calls the register() method of AbstractChannel.
childGroup.register(child).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
if (!future.isSuccess()) {
forceClose(child, future.cause());
}
}
});
} catch (Throwable t) {
forceClose(child, t);
}
}
register() method of AbstractChannel
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
if (eventLoop == null) {
throw new NullPointerException("eventLoop");
}
if (isRegistered()) {
promise.setFailure(new IllegalStateException("registered to an event loop already"));
return;
}
if (!isCompatible(eventLoop)) {
promise.setFailure(
new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
return;
}
AbstractChannel.this.eventLoop = eventLoop; // Initialize the eventLoop property of Channel
// If the new EventLoop is retrieved, its thread property is empty. The current thread is always the thread in the EventLoop in the bossGroup, so inEventLoop() returns false.
// If the old EventLoop is retrieved, its thread property is not empty. The current thread is always the thread in EventLoop in bossGroup, so inEventLoop() returns false.
if (eventLoop.inEventLoop()) { // Always return false. The current thread is always the thread in the EventLoop in the bossGroup. It must not be equal to the thread in any EventLoop in the workerGroup.
register0(promise);
} else {
try {
eventLoop.execute(new Runnable() { // This method is always executed
@Override
public void run() {
register0(promise);
}
});
} catch (Throwable t) {
logger.warn("Force-closing a channel whose registration task was not accepted by an event loop: {}",AbstractChannel.this, t);
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t);
}
}
}
public void execute(Runnable task) {
if (task == null) {
throw new NullPointerException("task");
}
boolean inEventLoop = inEventLoop();
addTask(task); // Put tasks in the queue
if (!inEventLoop) {
startThread(); // This method is called by either the new or the old EventLoop
if (isShutdown()) {
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && wakesUpForTask(task)) {
wakeup(inEventLoop);
}
}
private void startThread() {
if (state == ST_NOT_STARTED) { // If the old EventLoop is retrieved, its thread property is not empty, so its state property is not equal to st not started, so no new thread will be opened, and the registration task will wait for execution by the thread in the EventLoop
if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
boolean success = false;
try {
doStartThread(); // The new EventLoop is taken out. Its thread property is null, and its state is equal to st not started. Therefore, you need to start the thread, submit a task to the thread pool, and loop to handle the ready event of the Channel and execute all tasks saved in the task queue
success = true;
} finally {
if (!success) {
STATE_UPDATER.compareAndSet(this, ST_STARTED, ST_NOT_STARTED);
}
}
}
}
}
This is the core process for Netty to handle receive and read ready events
1. The client establishes a SocketChannel and connects.
2. The Selector in the EventLoop in the bossgroup listens that the ServerSocketChannel has a receive ready event.
3. receives the connection and finally calls the register() method of workerGroup. The method selects a EventLoop from the workGroup through the selector, and then calls the register() method of EventLoop.
4. finally, we call the register() method of Channel, initialize the eventLoop property of Channel, then register Channel into Selector in bossGroup EventLoop as a task, put it into Runnable instance, and then call EventLoop execute (EventLoop) method.
5. The execute() method puts the task into the task queue. If the new EventLoop is taken out, its thread attribute is empty. At this time, the thread will be started, and a task will be submitted to the thread pool. The loop will process the ready event of the Channel and execute all tasks saved in the task queue. If the old EventLoop is taken out, its thread attribute is not empty. Any The service waits to be executed by a thread in the EventLoop.
summary
1. After the NioEventLoopGroup instance is created, the size of its EventExecutor array and its stored EventLoop have been initialized.
2. Each NioEventLoop has a Selector, thread, Executor, Queue property. When a NioEventLoop instance is created, its thread property is still empty.
3. Each Channel will be bound to an EventLoop. Its eventLoop() method returns its bound EventLoop, and the Channel will be registered in the Selector of its bound EventLoop.
The register() method of 4.EventLoopGroup extracts a EventLoop from the EventLoopGroup through the selector and then calls the register() method of EventLoop.
5. The execute() method of the EventLoop will put the task into the task queue. If the inEventLoop() method returns false and its thread property is empty, a thread will be created and a task will be submitted to the thread pool (initialize the thread property in the EventLoop in the task, and then loop to handle the ready event of the Channel and execute all tasks saved in the task queue)