Java Network Programming and NIO Details 7: Brief Introduction to the Implementation Principle of NIO Selector in Linux

Keywords: Linux Java socket network

Talking about the Implementation Principle of Selector in Linux

From: https://www.jianshu.com/p/2b71ea919d49
Summary

Selector is the key class in NIO to realize I/O multiplexing. Selector achieves the goal of managing multiple Channel s through one thread, thus managing multiple network connections.
Channel represents this network connection channel. We can register Channel in Selector to manage it. A Channel can be registered in multiple different Selectors.
When Channel registers with Selector, it returns a SelectionKey object, which represents the relationship between the Channel and the Selector it registers with. And two important attributes are maintained in SelectionKey: interestOps, readyOps
interestOps is what we want Selector to listen to on Channel. We set the event we are interested in to this field so that when we find all the events we are interested in happening in the Channel during the selection operation, we will set the events we are interested in to readyOps again, so that we can know which events have happened for processing accordingly.

Important attributes in Selector

Three particularly important SelectionKey collections are maintained in Selector, which are

  • Keys: All SelectionKey represented by Channel registered with Selector exists in this collection. The addition of keys elements occurs when Channel registers with Selector.
  • Selected Keys: Each SelectionKey in this collection is its corresponding Channel that was checked during the last selection operation and at least one of the operations of interest in SelectionKey is ready to be processed. This set is a subset of keys.
  • Cancelled keys: SelectionKey that performs the cancel operation is put into the collection. This set is a subset of keys.

The following source code parsing illustrates the usefulness of the above three sets

Selector source code parsing

Next, we will go further into the implementation principle by explaining the process of using Selector.
Start with the simplest usage segment of Selector

        ServerSocketChannel serverChannel = ServerSocketChannel.open();
        serverChannel.configureBlocking(false);
        int port = 5566;          
        serverChannel.socket().bind(new InetSocketAddress(port));
        Selector selector = Selector.open();
        serverChannel.register(selector, SelectionKey.OP_ACCEPT);
        while(true){
            int n = selector.select();
            if(n > 0) {
                Iterator<SelectionKey> iter = selector.selectedKeys().iterator();
                while (iter.hasNext()) {
                    SelectionKey selectionKey = iter.next();
                    ......
                    iter.remove();
                }
            }
        }


1. Construction of Selector

Instance initialization of SocketChannel, ServerSocketChannel and Selector is implemented through the SelectorProvider class.

ServerSocketChannel.open();

    public static ServerSocketChannel open() throws IOException {
        return SelectorProvider.provider().openServerSocketChannel();
    }



SocketChannel.open();

    public static SocketChannel open() throws IOException {
        return SelectorProvider.provider().openSocketChannel();
    }

Selector.open();

    public static Selector open() throws IOException {
        return SelectorProvider.provider().openSelector();
    }


Let's take a closer look at SelectorProvider.provider()
    public static SelectorProvider provider() {
        synchronized (lock) {
            if (provider != null)
                return provider;
            return AccessController.doPrivileged(
                new PrivilegedAction<>() {
                    public SelectorProvider run() {
                            if (loadProviderFromProperty())
                                return provider;
                            if (loadProviderAsService())
                                return provider;
                            provider = sun.nio.ch.DefaultSelectorProvider.create();
                            return provider;
                        }
                    });
        }
    }

(1) If the "java.nio.channels.spi.SelectorProvider" property is configured, the SelectorProvider object corresponding to the value load of the property is loaded, and if the construction fails, an exception is thrown.
(2) If the provider class has been installed in the jar package visible to the system class loader and the source directory META-INF/services of the jar package contains a java.nio.channels.spi.SelectorProvider providing the class configuration file, the first class name in the file is loaded to construct the corresponding SelectorProvider object, if An exception is thrown if the build fails.
(3) If neither of the above two cases exists, return the default Selector Provider, sun. nio. ch. DefaultSelector Provider. create ().
(4) The method is then called, namely SelectorProvider.provider(). Returns the result of the first call.

Different systems correspond to different sun.nio.ch.DefaultSelector Provider

Here we look at sun. nio. ch. DefaultSelector Provider under linux

public class DefaultSelectorProvider {

    /**
     * Prevent instantiation.
     */
    private DefaultSelectorProvider() { }

    /**
     * Returns the default SelectorProvider.
     */
    public static SelectorProvider create() {
        return new sun.nio.ch.EPollSelectorProvider();
    }

}

As you can see, under linux system, sun. nio. ch. DefaultSelector Provider. create (); generates a sun. nio. ch. EPollSelector Provider type Selector Provider, which corresponds to the epoll of linux system.

Next, look at selector.open():
    /**
     * Opens a selector.
     *
     * <p> The new selector is created by invoking the {@link
     * java.nio.channels.spi.SelectorProvider#openSelector openSelector} method
     * of the system-wide default {@link
     * java.nio.channels.spi.SelectorProvider} object.  </p>
     *
     * @return  A new selector
     *
     * @throws  IOException
     *          If an I/O error occurs
     */
    public static Selector open() throws IOException {
        return SelectorProvider.provider().openSelector();
    }

After getting sun.nio.ch.EPollSelectorProvider, call the openSelector() method to build the Selector, where an EPollSelectorImpl object is built.

EPollSelectorImpl
class EPollSelectorImpl
    extends SelectorImpl
{

    // File descriptors used for interrupt
    protected int fd0;
    protected int fd1;

    // The poll object
    EPollArrayWrapper pollWrapper;

    // Maps from file descriptors to keys
    private Map<Integer,SelectionKeyImpl> fdToKey;
EPollSelectorImpl(SelectorProvider sp) throws IOException {
        super(sp);
        long pipeFds = IOUtil.makePipe(false);
        fd0 = (int) (pipeFds >>> 32);
        fd1 = (int) pipeFds;
        try {
            pollWrapper = new EPollArrayWrapper();
            pollWrapper.initInterrupt(fd0, fd1);
            fdToKey = new HashMap<>();
        } catch (Throwable t) {
            try {
                FileDispatcherImpl.closeIntFD(fd0);
            } catch (IOException ioe0) {
                t.addSuppressed(ioe0);
            }
            try {
                FileDispatcherImpl.closeIntFD(fd1);
            } catch (IOException ioe1) {
                t.addSuppressed(ioe1);
            }
            throw t;
        }
    }

The EPollSelectorImpl constructor completes:
(1) Construction of EPoll Array Wrapper, Epoll Array Wapper encapsulates Linux epoll-related system calls into a native method for Epoll Selector Impl.
(2) Register interruption events with epoll through EPoll Array Wrapper

    void initInterrupt(int fd0, int fd1) {
        outgoingInterruptFD = fd1;
        incomingInterruptFD = fd0;
        epollCtl(epfd, EPOLL_CTL_ADD, fd0, EPOLLIN);
    }

(3) fdToKey: Build a file descriptor-Selection Key Impl mapping table, and all the corresponding Selection Key and corresponding file descriptors registered with the channel of selector will be put into the mapping table.

EPollArrayWrapper

EPoll Array Wrapper completes the construction of epoll file descriptor and the encapsulation of epoll instruction manipulation in linux system. Maintain the result of each selection operation, the epoll_event array of the epoll_wait result.
EPoll Array Wrapper manipulates a local array of epoll_event structures on a linux system.

* typedef union epoll_data {
*     void *ptr;
*     int fd;
*     __uint32_t u32;
*     __uint64_t u64;
*  } epoll_data_t;
*
* struct epoll_event {
*     __uint32_t events;
*     epoll_data_t data;
* };

The data member of epoll_event (epoll_data_t data) contains the same data as the data set when the file descriptor is registered with epoll through epoll_ctl. Here data.fd is our registered file descriptor. So we have a valid file descriptor when dealing with events.

EPoll Array Wrapper encapsulates Linux epoll-related system calls into a native method for Epoll Selector Impl to use.

    private native int epollCreate();
    private native void epollCtl(int epfd, int opcode, int fd, int events);
    private native int epollWait(long pollAddress, int numfds, long timeout,
                                 int epfd) throws IOException;

The above three native methods correspond to three epoll-related system calls under Linux
    // The fd of the epoll driver
    private final int epfd;

     // The epoll_event array for results from epoll_wait
    private final AllocatedNativeObject pollArray;

    // Base address of the epoll_event array
    private final long pollArrayAddress;
    // Used to store associations between registered file descriptors and events whose registration awaits change. The epoll_wait operation is to detect whether events registered by the file descriptor here have occurred.
    private final byte[] eventsLow = new byte[MAX_UPDATE_ARRAY_SIZE];
    private final Map<Integer,Byte> eventsHigh = new HashMap<>();
    EPollArrayWrapper() throws IOException {
        // creates the epoll file descriptor
        epfd = epollCreate();

        // the epoll_event array passed to epoll_wait
        int allocationSize = NUM_EPOLLEVENTS * SIZE_EPOLLEVENT;
        pollArray = new AllocatedNativeObject(allocationSize, true);
        pollArrayAddress = pollArray.address();
    }

The EPoolArray Wrapper constructor creates the epoll file descriptor. An epoll_event array is constructed to store the results returned by epoll_wait.

Construction of Server Socket Channel

ServerSocketChannel.open();

Return the ServerSocket ChannelImpl object and construct the file descriptor of ServerSocket under the linux system.

    // Our file descriptor
    private final FileDescriptor fd;

    // fd value needed for dev/poll. This value will remain valid
    // even after the value in the file descriptor object has been set to -1
    private int fdVal;
    ServerSocketChannelImpl(SelectorProvider sp) throws IOException {
        super(sp);
        this.fd =  Net.serverSocket(true);
        this.fdVal = IOUtil.fdVal(fd);
        this.state = ST_INUSE;
    }


Register Server Socket Channel to Selector

serverChannel.register(selector, SelectionKey.OP_ACCEPT);
    public final SelectionKey register(Selector sel, int ops,
                                       Object att)
        throws ClosedChannelException
    {
        synchronized (regLock) {
            if (!isOpen())
                throw new ClosedChannelException();
            if ((ops & ~validOps()) != 0)
                throw new IllegalArgumentException();
            if (blocking)
                throw new IllegalBlockingModeException();
            SelectionKey k = findKey(sel);
            if (k != null) {
                k.interestOps(ops);
                k.attach(att);
            }
            if (k == null) {
                // New registration
                synchronized (keyLock) {
                    if (!isOpen())
                        throw new ClosedChannelException();
                    k = ((AbstractSelector)sel).register(this, ops, att);
                    addKey(k);
                }
            }
            return k;
        }
    }
    protected final SelectionKey register(AbstractSelectableChannel ch,
                                          int ops,
                                          Object attachment)
    {
        if (!(ch instanceof SelChImpl))
            throw new IllegalSelectorException();
        SelectionKeyImpl k = new SelectionKeyImpl((SelChImpl)ch, this);
        k.attach(attachment);
        synchronized (publicKeys) {
            implRegister(k);
        }
        k.interestOps(ops);
        return k;
    }

Constructing SelectionKey Object Representing the Relation between channel and selector
(2) Implement Register (k) Register channel into epoll
(3) k.interestOps(int) completes the following two operations:
(a) The registered interesting events and their corresponding file descriptions are stored in EvetsLow or EvetsHigh of the EPoll Array Wrapper object, which is used for the underlying implementation of epoll_wait.
b) At the same time, this operation will set the interestOps field of SelectionKey, which is used by our programmers.

EPollSelectorImpl. implRegister
    protected void implRegister(SelectionKeyImpl ski) {
        if (closed)
            throw new ClosedSelectorException();
        SelChImpl ch = ski.channel;
        int fd = Integer.valueOf(ch.getFDVal());
        fdToKey.put(fd, ski);
        pollWrapper.add(fd);
        keys.add(ski);
    }

(1) Put the FD (file descriptor) corresponding to channel and the corresponding Selection Key Impl into the fdToKey mapping table.
(2) Adding the fd (file descriptor) corresponding to channel to EPoll Array Wrapper, and forcing the initialization of the event of fd to be 0 (forcing the initial update event to be 0), because the event may exist in the previously cancelled registration. )
(3) Put selectionKey into the keys set.

Selection operation

There are three types of selection operations:
(1) select(): This method will block until at least one channel is selected (that is, the channel registered event occurs), unless the current thread interrupts or the selector's wakeup method is invoked.
(2) select(long time): This method is similar to select(), which can also cause blocking until at least one channel is selected (i.e., the channel registered event occurs), unless either of the following three situations occurs: a) the set timeout arrives; b) the current thread interrupts; c) the wakeup method of selector; Called
(3) selectNow(): This method will not block and will return immediately if no channel is selected.

Let's mainly look at the implementation of select(): int n = selector.select();
    public int select() throws IOException {
        return select(0);
    }

doSelect will eventually be invoked to EPollSelectorImpl

    protected int doSelect(long timeout) throws IOException {
        if (closed)
            throw new ClosedSelectorException();
        processDeregisterQueue();
        try {
            begin();
            pollWrapper.poll(timeout);
        } finally {
            end();
        }
        processDeregisterQueue();
        int numKeysUpdated = updateSelectedKeys();
        if (pollWrapper.interrupted()) {
            // Clear the wakeup pipe
            pollWrapper.putEventOps(pollWrapper.interruptedIndex(), 0);
            synchronized (interruptLock) {
                pollWrapper.clearInterrupted();
                IOUtil.drain(fd0);
                interruptTriggered = false;
            }
        }
        return numKeysUpdated;
    }

(1) Processing the logout selectionKey queue first
(2) Underlying epoll_wait operation
(3) Processing the cancelled selectionKey queue again
(4) Update the selected selectionKey

Let's start with processDeregisterQueue():
    void processDeregisterQueue() throws IOException {
        Set var1 = this.cancelledKeys();
        synchronized(var1) {
            if (!var1.isEmpty()) {
                Iterator var3 = var1.iterator();

                while(var3.hasNext()) {
                    SelectionKeyImpl var4 = (SelectionKeyImpl)var3.next();

                    try {
                        this.implDereg(var4);
                    } catch (SocketException var12) {
                        IOException var6 = new IOException("Error deregistering key");
                        var6.initCause(var12);
                        throw var6;
                    } finally {
                        var3.remove();
                    }
                }
            }

        }
    }

Remove the cancelled Selection Key from the cancelled Keys collection in turn, perform the cancellation operation, and remove the processed Selection Key from the cancelled Keys collection. When processDeregisterQueue() is executed, the cancelledKeys collection becomes empty.

    protected void implDereg(SelectionKeyImpl ski) throws IOException {
        assert (ski.getIndex() >= 0);
        SelChImpl ch = ski.channel;
        int fd = ch.getFDVal();
        fdToKey.remove(Integer.valueOf(fd));
        pollWrapper.remove(fd);
        ski.setIndex(-1);
        keys.remove(ski);
        selectedKeys.remove(ski);
        deregister((AbstractSelectionKey)ski);
        SelectableChannel selch = ski.channel();
        if (!selch.isOpen() && !selch.isRegistered())
            ((SelChImpl)selch).kill();
    }

Log-off completes the following operations:
(1) Remove the cancelled selectionKey from fdToKey (mapping table between file description and SelectionKey Impl)
(2) Remove the channel file descriptor represented by selectionKey from EPoll Array Wrapper
(3) Remove selectionKey from the keys collection so that the next selector.select() will no longer register the selectionKey to monitor in epoll
(4) The selectionKey will also be cancelled from the corresponding channel
Finally, if the corresponding channel is closed and no other selector is registered, the channel is closed.
After completing the operation, the cancelled SelectionKey will not appear first in any of the three sets of keys, selectedKeys and cancelKeys.


Then let's look at EPoll Array Wrapper. poll (timeout):
    int poll(long timeout) throws IOException {
        updateRegistrations();
        updated = epollWait(pollArrayAddress, NUM_EPOLLEVENTS, timeout, epfd);
        for (int i=0; i<updated; i++) {
            if (getDescriptor(i) == incomingInterruptFD) {
                interruptedIndex = i;
                interrupted = true;
                break;
            }
        }
        return updated;
    }

The updateRegistrations() method registers events that have been registered with the selector (eventsLow or eventsHigh) into the linux system by calling epollCtl(epfd, opcode, fd, events).
Here epollWait calls the epoll_wait method at the bottom of linux and returns the number of entries triggered by events during epoll_wait.

Look at updateSelectedKeys():
    private int updateSelectedKeys() {
        int entries = pollWrapper.updated;
        int numKeysUpdated = 0;
        for (int i=0; i<entries; i++) {
            int nextFD = pollWrapper.getDescriptor(i);
            SelectionKeyImpl ski = fdToKey.get(Integer.valueOf(nextFD));
            // ski is null in the case of an interrupt
            if (ski != null) {
                int rOps = pollWrapper.getEventOps(i);
                if (selectedKeys.contains(ski)) {
                    if (ski.channel.translateAndSetReadyOps(rOps, ski)) {
                        numKeysUpdated++;
                    }
                } else {
                    ski.channel.translateAndSetReadyOps(rOps, ski);
                    if ((ski.nioReadyOps() & ski.nioInterestOps()) != 0) {
                        selectedKeys.add(ski);
                        numKeysUpdated++;
                    }
                }
            }
        }
        return numKeysUpdated;
    }

This method retrieves event-triggered SelectionKeyImpl objects from EPoll Array Wrapper poll Wrapper and fdToKey (build file descriptor-SelectorKeyImpl mapping table), and then places SelectionKeyImpl into the selectedKey collection (selectionKey collection with event trigger, which can be passed through selector.selectedKeys() method. Get) in, that is, selectedKeys. And reset the relevant readyOps values in Selection Key Impl.
But here are two points to note:
(1) If SelectionKey Impl already exists in the selected keys collection and finds that the triggered event already exists in readyOps, it will not make numKeys Updated++; this will make it impossible for us to know the change of the event. This explains why we remove the Selection Key from the selectedKey collection every time we retrieve it from the selectedKey collection, so that the selectionKey can be correctly placed into the selectedKey collection and correctly notified to the caller when an event triggers. Furthermore, if you do not remove the processed SelectionKey from the selectedKey collection, the next time a new event arrives, you will traverse the SelectionKey again when traversing the selectedKey collection, which is likely to make a mistake. For example, if the corresponding SelectionKey is not removed from the selectedKey set after the OP_ACCEPT event is processed, the next time the selectedKey set is traversed, the corresponding ServerSocketChannel.accept() will return an empty (null) SocketChannel.

(2) If I/O events occurring in channel are not of current interest to SelectionKey, the SelectionKey Impl will not be placed in the selected keys collection, nor will numKeys Updated++ be made available.


Wechat Public [Java Technology Jianghu], a technology station for an Ali Java engineer. Respond to "Java" after paying attention to the public number, you can get free learning materials such as Java Foundation, Advancement, Projects and Architects, as well as popular technology learning videos such as database, distributed, micro-services, which are rich in content and take into account the principles and practices. In addition, the author's original Java learning guide and interview guide for Java programmers will also be presented. Equivalent dry goods resources)

Posted by leeue on Fri, 09 Aug 2019 22:41:20 -0700