Epic strongest textbook style "NIO and Netty programming"
1.1 Overview
The full name of java.nio, java non-blocking IO, refers to the new API that JDK1.4 has started to offer.Beginning with JDK1.4, Java has provided a series of new features for improved input/output, also known as NIO (New IO), with many new classes for processing input and output, which are placed under java.nio packages and subpackages and are aligned with each other http://java.io Many of the classes in the package are overwritten and new classes are added to meet NIO's capabilities.
NIO and BIO have the same purpose and function, but they are implemented in a completely different way. BIO processes data in streaming mode, whereas NIO processes data in block mode, block I/O is much more efficient than stream I/O.In addition, NIO is non-blocking, which is different from BIO in that it can provide a non-blocking, highly scalable network.
NIO has three main core parts: Channel, Buffer, Selector.Traditional BIO s operate on byte and character streams, whereas NIOs operate on Channel and Buffer, where data is always read from or written to the channel.Selector is used to listen for events across multiple channels (e.g., connection open, data arrival).So you can listen on multiple data pipelines using a single thread.
1.2 File IO
1.2.1 Overview and Core API s
Buffer: A container, in fact, is a special array. Buffer objects have built-in streaming mechanisms that track and record changes in the state of the buffer.Channel provides access to data from files and networks, but data read or written must go through Buffer, as shown in the following figure:
In NIO, Buffer is a top-level parent class, it is an abstract class, and the common subclasses of Buffer are:
ByteBuffer, storing byte data to buffer
ShortBuffer, storing string data to buffer
CharBuffer, stores character data to buffer
IntBuffer, stores integer data to buffer
LongBuffer, stores long integer data to buffer
DoubleBuffer, store decimal to buffer
FloatBuffer, store decimal to buffer
For the basic data types in Java, there is a specific Buffer type that corresponds to it. The most common natural type is the ByteBuffer class (binary data), whose main methods are as follows:
The ByteBuffer class (binary data) has the following main methods:
public abstract ByteBuffer put(byte[] b); stores byte data to buffer
public abstract byte[] get(); get byte data from buffer
public final byte[] array(); converts buffer data to byte array
public static ByteBuffer allocate(int capacity); sets the initial capacity of the buffer
public static ByteBuffer wrap(byte[] array); use an existing array in the buffer
public final Buffer flip(); flip buffer, reset position to initial position
Channel: A stream in BIO, such as a FileInputStream object, used to establish a connection to a destination (file, network socket, hardware device, etc.), but it is important to note that streams in BIO are one-way, for example, FileInputStream objects can only read data, while streams in NIO can only read data.Channels are bidirectional and can be used for both read and write operations.Common Channel classes are FileChannel, DatagramChannel, ServerSocketChannel, and SocketChannel.FileChannel is used for data reading and writing of files, DatagramChannel, ServerSocketChannel and SocketChannel.FileChannel is used for data reading and writing of files, DatagramChannel for data reading and writing of UDP, ServerSocketChannel and SocketChannel for data reading and writing of TCP.
The FileChannel class, which is mainly used for IO operations on local files, is as follows:
public int read (ByteBuffer dst), read the data and put it in the buffer
public int write(ByteBuffer src), which writes data from the buffer to the channel
public long transferFrom(ReadableByteChannel src,long position,long count), replicates data from the target channel
public long transferTo(long position,long count,WritableByteChannel target) copies data from the current channel to the target channel
1.2.2 Cases
1. Write data to a local file
@Test
public void contextLoads() throws Exception {
String str = "hello,nio,I am a cereal";
// Create Output Stream
FileOutputStream fileOutputStream = new FileOutputStream("basic.txt");
// Get a channel from the stream
FileChannel fileChannel = fileOutputStream.getChannel();
// Provide a buffer
ByteBuffer allocate = ByteBuffer.allocate(1024);
// Store data in buffer
allocate.put(str.getBytes());
// When data is written to the buffer, the pointer points to the last row of data, and when the buffer writes to the output in the channel, it writes from the last row of data.
// This will result in an empty buffer with no data remaining to write 1024.So you need to flip the buffer and reset the position to the initial position
allocate.flip();
// Writes buffers to channels responsible for writing data to files
fileChannel.write(allocate);
// Close the output stream because the channels are created by the output stream, so they are closed together
fileOutputStream.close();
}
Channels in NIO are acquired from the output stream object through the getChannel method and are bidirectional, readable and writable.Before writing data to a channel, the data must be saved to ByteBuffer through the put method, and then written through the channel write.Before writing, you need to invoke the flip method to flip the buffer and position the interior to the initial position so that you can write all the data to the channel only when you write the data next.The results are as follows:
@Test // Read data from local files public void test2() throws Exception { File file = new File("basic.text"); // 1. Create input stream FileInputStream fis = new FileInputStream(file); // 2. Get a channel FileChannel fc = fis.getChannel(); // 3. Prepare a buffer ByteBuffer buffer = ByteBuffer.allocate((int)file.length()); // 4. Read data from the channel and store it in the buffer fc.read(buffer); System.out.println(new String(buffer.array)); // 5. Close fis.close(); } @Test // File Copy Using NIO public void test3() throws Exception { //1. Create two streams FileInputStream fis = new FileInputStream("basic.text"); FileOutputStream fos = new FileOutputStream("c:\\test\\basic.text"); // 2. Get two channels FileChannel sourceFc = fis.getChannel(); FileChannel destFc = fos.getChannel(); //3. Copy destFc.transferFrom(sourceFc,0,sourceFc.size()); //4. Close fis.close(); fos.close(); }
1.3 Network IO
1.3.1 Overview and Core API s
The FileChannel used in the previous file IO does not support non-blocking operations. Learning NIO mainly implements network IO. Network Channel in Java NIO is the implementation of non-blocking IO. Based on event-driven, it is ideal for situations where the server needs to maintain a large number of connections, but the amount of data exchange is small, such as some instant messaging.Message service is waiting.
Writing a Socket server in Java usually has the following modes:
A client connection uses a single thread, with the advantages of simple programming and the disadvantages of having many connections and allocating many threads, the server may crash due to exhaustion of resources.
Give each client connection to a pool of connections with a fixed number of threads, with the advantage that the program is relatively simple to write and can handle a large number of connections.Disadvantages: Threads are very expensive, and if there are many connections, queuing can become more severe.
Using Java's NIO, it is handled in a non-blocking IO manner.This mode can handle a large number of client connections with a single thread.
1.Selector, a selector that detects whether events occur on multiple registered channels, fetches the events, and processes the responses accordingly for each event.This allows you to manage multiple channels with a single thread, that is, multiple connections.This allows functions to be called for read-write only when the connection has a real read-write event, greatly reducing overhead, eliminating the need to create a thread for each connection, maintaining multiple threads, and avoiding the overhead of context switching between multiple threads.
Common methods for this class are as follows:
public static Selector open(), get a selector object
public int select(long timeout), which monitors all registered channel s, joins the corresponding SelectionKey to the internal group and returns when there are registered IO operations available, and the parameters are used to set the timeout
public Set selectedKeys(), Gets all SelectionKey from the internal collection
2. SelectionKey, which represents the registration relationship between Selector and serverSocketChannel, has four types:
int OP_ACCEPT: A new network connection is available to accept with a value of 16
int OP_CONNECT: Represents that the connection has been established, with a value of 8
int OP_READ and int OP_WRITE: Represents read and write operations with values of 1 and 4
Common methods for this class are as follows:
public abstract Selector selector(), get the Selector object associated with it
public abstract SelectorChannel channel(), get the channel associated with it
public final Object attachment(), get the shared data associated with it
public abstract SelectionKey interestOps(int ops), setting or changing listening events
public final boolean isAcceptable(), can you accept
public final boolean isReadable(), is it readable
public final boolean isWritable(), is it writable
3. ServerSocketChannel, which is used to listen on the server side for new client Socket connections, is commonly used as follows:
public static ServerSocketChannel open(), get a ServerSocketChannel channel
public final ServerSocketChannel bind(SocketAddress local), set the server-side port number
Public final Selectable Channel configureBlocking (boolean block), set blocking or non-blocking mode, value false means non-blocking mode
public SocketChannel accept(), accepts a connection and returns the channel object representing that connection
public final SelectionKey register(Selector sel,int ops), register a selector, and set up listening events
4. SocketChannel, network IO channel, responsible for read and write operations.NIO always writes data from buffers to channels or reads data from channels to buffers.Common methods are as follows:
public static SocketChannel open(), get a SocketChannel channel
Public final Selectable Channel configureBlocking (boolean block), set blocking or non-blocking mode, value false means non-blocking mode
public boolean connect(SocketAddress remote), connect to server
public boolean finishConnect(), if the above method fails to connect, the next step is to complete the connection using this method
public int write(ByteBuffer src), write data to channel
public int read(ByteBuffer dst), reading data from a channel
public final SelectionKey register(Selector sel,int ops,Object att), registers a selector and sets the listen event, and the last parameter sets the shared data
public final void close(), closing the channel
3.4 AIO Programming
JDK1.7 introduced Asynchronous I/O, AIO.Two modes are commonly used in I/O programming: Reactor and Proctor.Java's NIO is the Reactor, and when an event triggers, the server is notified and handled accordingly.
AIO, NIO 2.0, is called asynchronous non-blocking IO.AIO introduces the concept of asynchronous channel and uses Proactor mode to simplify programming. A valid request starts a thread. It is characterized by notifying the server program to start a thread after the completion of the operating system. It is generally suitable for applications with a large number of connections and a long connection time.
3.5 IO Comparison Summary
There are usually several ways of IO, synchronous blocked BIO, synchronous non-blocked IO, asynchronous non-blocked IO.
The BIO approach is suitable for architectures with relatively small and fixed number of connections. It requires more server resources and is confined to applications. It was the only option before JDK1.4, but the program is intuitive and easy to understand.
NIO is suitable for architectures that connect more data and have a shorter connection (light operation), such as chat servers, which are limited to concurrent applications, complex programming, and JDK1.4 support.
AIO is suitable for architectures with many connections and long connections (re-operation), such as album servers, fully invoking OS to participate in concurrent operations, complex programming, and JDK7 support.
4.1 Overview
Netty is a Java open source framework provided by JBOSS.Netty provides an asynchronous, event-driven network application framework and tools for the rapid development of high-performance, highly reliable network IO programs.
Netty is a NIO-based network programming framework. Using Netty can help you develop a network application quickly and easily, which is equivalent to simplifying and streamlining the development process of NIO.As the most popular NIO framework, Netty is widely used in the Internet, big data distributed computing, games, communications and other fields. Netty is used in the well-known Elasticsearch, Dubbo framework.
4.2 Netty Overall Design
4.2.1 Thread Model
1. Single Thread Model
The server side uses a single thread to multiplex all IO operations (including connections, reads, writes, etc.). The coding is simple and the cleaning is clear, but if the number of client connections is large, it will not be supported. The NIO case before us belongs to this model.
2. Thread pool model
The server side uses a thread to handle client connection requests, and a thread group to handle IO operations.The model works well in most scenarios.
3.Netty Model
Netty abstracts two sets of thread pools, BossGroup is responsible for receiving client connections, and WorkderGroup is responsible for network read and write operations.NioEventLoop represents a thread that continuously loops through processing tasks, and each NioEventLoop has a selector to listen for socket network channels bound to it.NioEventLoop is designed internally to be serialized and sent from message reading - "decoding -" processing - "encoding -", always under the responsibility of the IO thread NioEventLoop.
Include multiple NioEventLoops under a NioEventLoopGroup
Each NioEventLoop contains a Selector, a taskQueue
Each NioChannel will only be bound to a unique NioEventLoop
Each NioChannel is bound to its own ChannelPipeline
4.2.2 Asynchronous Model
FUTURE, CALLBACK, and HANDLER
Netty's asynchronous model is built on futures and callback s.The core idea of Future is that assuming a method, fun, the calculation process can be time consuming and waiting for funs to return is obviously not appropriate.You can then immediately return to a Future when a fun is called and subsequently monitor the processing of the method fun through Future.
When programming with Netty, interception and conversion of inbound and outbound data require only a callback or a future.This makes chain-based operations simple, efficient, and useful for writing reusable, generic code.The goal of the Netty framework is to separate your business logic from your network-based application code.
4.3 Core API
- ChannelHandler and its implementation class
The ChannelHandler interface defines a number of event handling methods that we can implement specific business logic by overriding.The API relationship is illustrated below:
Customizing a Handler class to inherit the ChannelInboundHandlerAdapter and then implementing business logic by rewriting the appropriate method generally requires rewriting the following methods:
public void channelActive(ChannelHandlerContext ctx), channel ready event
public void channelRead(ChannelHandlerContext ctx,Object msg), channel read data events
public void channelReadComplete(ChannelHandlerContext ctx), data read complete event
public void exceptionCaught(ChannelHandlerContext ctx,Throwable cause), channel exception event
- Pipeline and Channel Pipline
ChannelPipeline is a collection of Handler s that handle and intercept events and operations on inbound or outbound, equivalent to a chain through Netty.
ChannelPipeline addFirst(ChannelHandler...handlers), add a business processing class (handler) to the first location in the chain
Channel Pipeline addLast (ChannelHandler...handlers), which adds a business processing class (handler) to the last location in the chain
- ChannelHandlerContext
This is the event handler context object, the actual processing node in the Pipeline chain.Each processing node, ChannelHandlerContext, contains a specific event handler, ChannelHandlerContext, which also binds information about the corresponding pipeline and Channel, making it easy to call ChannelHandler.
Common methods are as follows:
ChannelFuture close(), closing channel
ChannelOutboundInvoker flush(), refresh
ChannelFuture writeAndFlush(Object msg), which writes data to the next ChanelHandler of the current ChannelHandler in ChannelPipeline to start processing (outbound) - ChannelOption
Netty generally needs to set the ChannelOption parameter after creating a Channel instance.ChannelOption is a standard parameter for Socket s, not Netty's.Common parameter configurations are:
- ChannelOption.SO_BACKLOG
Corresponds to the backlog parameter in the TCP/IP protocol listen function, which is used to initialize the size of the server connectable queue.The server processes client connection requests sequentially, so only one client connection can be processed at a time.When more than one client comes, the server queues client connection requests that it cannot handle, and the backlog parameter specifies the size of the queue d. - ChannelOption.SO_KEEPALIVE
Keep the connection active.
- ChannelFuture
Represents the result of an asynchronous I/O operation in Channel. All I/O operations in Netty are asynchronous. The I/O call returns directly. The caller cannot get the result immediately, but the processing state of the I/O operation can be obtained through ChannelFuture.Common methods are as follows:
Channel channel(), returns the channel that is currently operating on IO
ChannelFuture sync(), waiting for the asynchronous operation to complete - EventLoopGroup and its implementation class NioEventLoopGroup
EventLoopGroup is the abstraction of a set of EventLoops. To make better use of multi-core CPU resources, Netty typically has multiple EventLoops working simultaneously, and each EventLoop maintains a Selector instance.
EventLoopGroup provides the next interface, which allows you to get one of the EventLoops from a group to process tasks according to certain rules.In Netty server-side programming, we typically need to provide two EventLoopGroups, such as BossEventLoopGroup and WorkderEventLoopGroup.
Usually a service port, a ServerSocketChannel, corresponds to a Selector and an EventLoop thread.BossEventLoop is responsible for receiving client connections and handing the SocketChannel to WorkerEventLoopGroup for IO processing, as shown in the following figure:
BossEventLoopGroup is usually a single-threaded EventLoop. EventLoop maintains a Selector instance registered with ServerSocketChannel. BossEventLoop continuously polls Selector to isolate connection events, usually OP_ACCEPT events, and then delivers the received SocketChannel to WorkderEventLoopGroup, WorkderEventLoopGroup will have next select one of the EventLoopGroups to register this SocketChannel with its maintained Selector and process its subsequent IO events.
Common methods are as follows:
public NioEventLoopGroup(), construction method
Public Future<?> shutdownGracefully (), disconnect, close thread
- ServerBootstrap and Bootstrap
ServerBootStrap is the server-side startup assistant in Netty, through which various server-side configurations can be completed; Bootstrap is the client-side startup assistant in Netty, through which various client configurations can be completed.Common methods are as follows:
public ServerBootstrap group(EventLoopGroup parentGroup,EventLoopGroup childGroup), which is used on the server side to set up two EventLoops
Public B group (EventLoopGroup), which is used by clients to set up an EventLoop
Public B channel (Class<? Extends C> channelClass), which is used to set up a server-side channel implementation
public B option(ChannelOption option,T value) to add configuration to ServerChannel
public ServerBootstrap childOption(ChannelOption childOption,T value), which is used to add configuration to the received channel
Public ServerBootstrap childHandler, which sets the business processing class (custom handler)
public ChannelFuture bind(int inetPort), which is used on the server side to set the port number occupied
public ChannelFuture connect(String inetHost,int inetPort), which is used on the client side to connect to the server side - Unpooled class
This is a tool class provided by Netty specifically for manipulating buffers using the following common methods:
public static ByteBuf copiedBuffer(CharSequence string, Charset charset), returns a ByteBuf object (similar to a ByteBuffer object in Nio) with the given data and character encoding
Starter Cases:
<dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> <version>4.1.25.Final</version> </dependency> package com.example.testdemo.netty; import io.netty.bootstrap.ServerBootstrap; import io.netty.channel.*; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioServerSocketChannel; public class NettyServer { public static void main(String[] args) throws InterruptedException { // 1. Create a thread group to receive client connections EventLoopGroup bossGroup = new NioEventLoopGroup(); // 2. Create a thread group to handle network operations EventLoopGroup workerGroup = new NioEventLoopGroup(); // 3. Create a server-side startup assistant to configure parameters ServerBootstrap serverBootstrap = new ServerBootstrap(); // 4. Set up two thread groups serverBootstrap.group(bossGroup, workerGroup) // 5.Implementation of Server-side Channel Using NioServerSocketChannel .channel(NioServerSocketChannel.class) // 6. Set the number of connections waiting in the thread queue .option(ChannelOption.SO_BACKLOG, 128) // 7. Stay connected .childOption(ChannelOption.SO_KEEPALIVE, true) // 8.Create a channel initialization object .childHandler(new ChannelInitializer<SocketChannel>() { // 9.Add a custom handler class to the pipline chain @Override protected void initChannel(SocketChannel channel) throws Exception { channel.pipeline().addLast(new NettyServerHandler()); } }); System.out.println("...Server ready...."); // 10. Bind port, bind method asynchronous, sync synchronous blocking ChannelFuture sync = serverBootstrap.bind(9999).sync(); System.out.println("...server start"); // 11.Close channel, close thread group sync.channel().closeFuture().sync(); bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully(); } } package com.example.testdemo.netty; import io.netty.buffer.ByteBuf; import io.netty.buffer.Unpooled; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelInboundHandlerAdapter; import io.netty.util.CharsetUtil; public class NettyServerHandler extends ChannelInboundHandlerAdapter { /** * Read Data Events * * @param ctx * @param msg * @throws Exception */ @Override public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception { System.out.println("Server : " + ctx); ByteBuf byteBuf = (ByteBuf)msg; System.out.println("Message from client:" + byteBuf.toString(CharsetUtil.UTF_8)); } /** * Data Read Complete Event * * @param ctx * @param cause * @throws Exception */ @Override public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { ctx.close(); } @Override public void channelReadComplete(ChannelHandlerContext ctx) throws Exception { ctx.writeAndFlush(Unpooled.copiedBuffer("Just run out of money", CharsetUtil.UTF_8)); } } package com.example.testdemo.netty; import io.netty.bootstrap.Bootstrap; import io.netty.channel.ChannelFuture; import io.netty.channel.ChannelInitializer; import io.netty.channel.EventLoopGroup; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioSocketChannel; public class NettyClient { public static void main(String[] args) throws InterruptedException { // 1. Create a thread group EventLoopGroup group = new NioEventLoopGroup(); // 2.Create client startup assistant to complete Hong Kong configuration Bootstrap bootstrap = new Bootstrap(); // 3. Set Thread Group bootstrap.group(group) // 4. Set the implementation class for the client channel .channel(NioSocketChannel.class) // 5. Create a channel initialization object .handler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel socketChannel) throws Exception { // 6. Add a custom handler to the pipline chain socketChannel.pipeline().addLast(new NettyClientHandler()); } }); // 7. Start the client to connect to the server asynchronously and non-blocking. Connect is asynchronous, it immediately returns a future object, sync is synchronously blocked to wait for the main thread System.out.println("...Client is ready ..."); ChannelFuture sync = bootstrap.connect("127.0.0.1", 9999).sync(); // 8. Close connections asynchronously and non-blocking sync.channel().closeFuture().sync(); } } package com.example.testdemo.netty; import io.netty.buffer.ByteBuf; import io.netty.buffer.Unpooled; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelInboundHandlerAdapter; import io.netty.util.CharsetUtil; /** * Client Business Processing Class */ public class NettyClientHandler extends ChannelInboundHandlerAdapter { @Override public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception { ByteBuf byteBuf = (ByteBuf)msg; System.out.println("Message from server side: " + byteBuf.toString(CharsetUtil.UTF_8)); } /** * Channel Ready Event * * @param ctx * @throws Exception */ @Override public void channelActive(ChannelHandlerContext ctx) throws Exception { System.out.println("Client : " + ctx); ctx.writeAndFlush(Unpooled.copiedBuffer("Boss, pay back the money", CharsetUtil.UTF_8)); } }
Chat case:
package com.example.testdemo.netty; import io.netty.bootstrap.ServerBootstrap; import io.netty.channel.ChannelFuture; import io.netty.channel.ChannelInitializer; import io.netty.channel.ChannelOption; import io.netty.channel.EventLoopGroup; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioServerSocketChannel; import io.netty.handler.codec.string.StringDecoder; import io.netty.handler.codec.string.StringEncoder; /** * Chat Program Server Side */ public class ChatServer { /** * Server-side port number */ private int port; public ChatServer(int port) { this.port = port; } public void run() throws Exception { EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup(); try { ServerBootstrap serverBootstrap = new ServerBootstrap(); serverBootstrap.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .option(ChannelOption.SO_BACKLOG, 128) .childOption(ChannelOption.SO_KEEPALIVE, true) .childHandler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel socketChannel) throws Exception { socketChannel.pipeline() // Add a decoder to the pipeline chain .addLast("decoder", new StringDecoder()) // Add an encoder to the pipeline chain .addLast("encoder", new StringEncoder()) // Add a custom handler (Business Processing Class) to the pipline chain .addLast(new ChatServerHandler()); } }); System.out.println("Netty chat Server Start."); ChannelFuture channelFuture = serverBootstrap.bind(port).sync(); channelFuture.channel().closeFuture().sync(); } finally { workerGroup.shutdownGracefully(); bossGroup.shutdownGracefully(); System.out.println("Netty chat Server Close..."); } } public static void main(String[] args) throws Exception { new ChatServer(9999).run(); } } package com.example.testdemo.netty; import io.netty.channel.Channel; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.SimpleChannelInboundHandler; import java.util.ArrayList; import java.util.List; /** * Customize a server-side business processing class */ public class ChatServerHandler extends SimpleChannelInboundHandler<String> { private static List<Channel> channels = new ArrayList<>(); /** * Read data * * @param channelHandlerContext * @param s * @throws Exception */ @Override protected void channelRead0(ChannelHandlerContext channelHandlerContext, String s) throws Exception { Channel inChannel = channelHandlerContext.channel(); channels.forEach(channel -> { if (channel != inChannel) { channel.writeAndFlush("[" + inChannel.remoteAddress().toString().substring(1) + "]" + "Say:" + s + "\n"); } }); } /** * Channel ready * * @param ctx * @throws Exception */ @Override public void channelActive(ChannelHandlerContext ctx) throws Exception { Channel channel = ctx.channel(); channels.add(channel); System.out.println("[Server] : " + channel.remoteAddress().toString().substring(1) + "Go online"); } /** * Channel not ready * * @param ctx * @throws Exception */ @Override public void channelInactive(ChannelHandlerContext ctx) throws Exception { Channel channel = ctx.channel(); channels.remove(channel); System.out.println("[Server] : " + channel.remoteAddress().toString().substring(1) + "Off-line"); } } package com.example.testdemo.netty; import io.netty.bootstrap.Bootstrap; import io.netty.channel.*; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioSocketChannel; import io.netty.handler.codec.string.StringDecoder; import io.netty.handler.codec.string.StringEncoder; import java.util.Scanner; /** * Chat Program Client */ public class ChatClient { /** * Server-side IP address */ private final String host; /** * Server-side ports and */ private final int port; public ChatClient(String host, int port) { this.host = host; this.port = port; } public void run() { EventLoopGroup group = new NioEventLoopGroup(); try { Bootstrap bootstrap = new Bootstrap(); bootstrap.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel socketChannel) throws Exception { ChannelPipeline pipeline = socketChannel.pipeline(); // Add a decoder to the pipeline chain pipeline.addLast("decoder", new StringDecoder()); // Add an encoder to the pipeline chain pipeline.addLast("encoder", new StringEncoder()); // Add a custom handler (business process class) to the pipeline chain pipeline.addLast(new ChatClientHandler()); } }); ChannelFuture channelFuture = bootstrap.connect(host, port).sync(); Channel channel = channelFuture.channel(); System.out.println("----" + channel.localAddress().toString().substring(1) + "----"); Scanner scanner = new Scanner(System.in); while (scanner.hasNextLine()) { String nextLine = scanner.nextLine(); channel.writeAndFlush(nextLine + "\r\n"); } } catch (Exception e) { e.printStackTrace(); } finally { group.shutdownGracefully(); } } public static void main(String[] args) { new ChatClient("127.0.0.1", 9999).run(); } } package com.example.testdemo.netty; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.SimpleChannelInboundHandler; /** * Customize a client business processing class */ public class ChatClientHandler extends SimpleChannelInboundHandler<String> { @Override protected void channelRead0(ChannelHandlerContext channelHandlerContext, String s) throws Exception { System.out.println(s.trim()); } }
4.6 Encoding and decoding
4.6.1 Overview
When writing network applications, you need to pay attention to codec (codec), because the data is transmitted in the network as binary bytes? Data, and we often get the target data is not byte? Data.Therefore, encoding is required when sending data and decoding is required when receiving data.
There are two components of codec: decoder and encoder.Encoder is responsible for converting business data to byte code data, and decoder is responsible for converting byte code data to business data.
In fact, the serialization technology of java can be used as codec, but it has too many hard injuries:
- Cannot cross-language, this should be the most fatal problem in Java serialization.
- Serialization is too large, more than five times as large as binary encoding.
- Serialization performance is too low.
Because Java serialization is hardened too much, Netty itself provides some codec s, as shown below:
Netty provides decoders: - StringDecoder, decoding string data
- ObjectDecoder, decoding Java objects
Netty provides decoders: - StringEncoder, encoding string data
- ObjectEncoder, encoding Java objects
Netty's own ObjectDecoder and ObjectEncoder can be used to encode and decode POJP objects or various business objects, but Java serialization technology is still used internally, so it is not recommended.
4.6.2 Google's Proobuf
Protobuf is an open source project published by Google, full name of Google Protocol Buffers, specific as follows:
- Support for cross-platform, multi-language (support for most languages such as C++, C#, Java, Python, etc.)
- High performance, high reliability
- Code can be automatically generated using the protobuf compiler, which describes the definition of a class using a.protp file, and then automatically generates a.java file from the.proto using the protoc.exe compiler.
Currently, when developing with Netty, Protobuf is often used in conjunction with codec (codec) as follows:
<!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.9.1</version>
</dependency>
Generate a java class from this file using protoc.exe as follows:
Run the command in the protoc.exe root directory: protoc --java_out=. Book.pro
5.1 Custom RPC
Summary:
RPC (Remote Procedure Call), a remote procedure call, is a technology that requests services from a remote computer program over a network without having to know the underlying network implementation.Common RPC frameworks are: Dubbo, Grpc.
- Service Consumer House (client) calls services locally
- client stub is responsible for encapsulating methods, parameters, etc. into a message body that can be transmitted when it receives a call
- client stub encodes messages and sends them to the server
- server stub decodes when it receives a message
- server stub calls local services based on decoding results
- Local service execution and return results to server stub
- server stub will return the import result to encode and send to the consumer
- client stub receives messages and decodes them
- Service consumer (client) gets results
The goal of RPC is to encapsulate these 2-8 steps so that users can complete remote service calls as if they were calling a local method without having to deal with these details.
5.2 Design and Implementation
5.2.1 Result Design
- Client (caller of service): two interfaces + a test class containing the main method
- Client Stub: a client proxy class + a client business processing class
- Server (provider of service): two interfaces + two implementation classes
- Server Stub: a network processing server + a server business processing class
Note: The service caller's interface must be consistent with the service provider's interface (package paths can be inconsistent)
The ultimate goal is to call methods in HelloRPCImpl or HelloNettyImpl remotely in TestNettyRPC
package com.example.testdemo.rpc; import io.netty.bootstrap.ServerBootstrap; import io.netty.channel.*; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioServerSocketChannel; import io.netty.handler.codec.serialization.ClassResolvers; import io.netty.handler.codec.serialization.ObjectDecoder; import io.netty.handler.codec.serialization.ObjectEncoder; public class NettyRPCServer { private int port; public NettyRPCServer(int port) { this.port = port; } public void start() { EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup(); try { ServerBootstrap serverBootstrap = new ServerBootstrap(); serverBootstrap.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .option(ChannelOption.SO_BACKLOG, 128) .childOption(ChannelOption.SO_KEEPALIVE, true) .localAddress(port) .childHandler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel ch) throws Exception { ChannelPipeline pipeline = ch.pipeline(); // Encoder pipeline.addLast("encoder", new ObjectEncoder()); // Decoder pipeline.addLast("decoder", new ObjectDecoder(Integer.MAX_VALUE, ClassResolvers.cacheDisabled(null))); // Server-side Business Processing Class pipeline.addLast(new InvokerHandler()); } }); ChannelFuture future = serverBootstrap.bind(port).sync(); System.out.println("...Server is ready..."); future.channel().closeFuture().sync(); } catch (Exception e) { bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully(); } } public static void main(String[] args) { new NettyRPCServer(9999).start(); } } package com.example.testdemo.rpc; import io.netty.bootstrap.Bootstrap; import io.netty.channel.ChannelFuture; import io.netty.channel.ChannelInitializer; import io.netty.channel.ChannelPipeline; import io.netty.channel.EventLoopGroup; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioSocketChannel; import io.netty.handler.codec.serialization.ClassResolvers; import io.netty.handler.codec.serialization.ObjectDecoder; import io.netty.handler.codec.serialization.ObjectEncoder; import java.lang.reflect.InvocationHandler; import java.lang.reflect.Method; import java.lang.reflect.Proxy; public class NettyRPCProxy { /** * Create proxy object from interface * * @param target * @return */ public static Object create(Class target) { return Proxy.newProxyInstance(target.getClassLoader(), new Class[]{target}, new InvocationHandler() { @Override public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { // Encapsulate classinfo ClassInfo classInfo = new ClassInfo(); classInfo.setClassName(getClass().getName()); classInfo.setMethodName(method.getName()); classInfo.setObjects(args); classInfo.setTypes(method.getParameterTypes()); // Start sending data with Netty EventLoopGroup group = new NioEventLoopGroup(); ResultHandler resultHandler = new ResultHandler(); try { Bootstrap bootstrap = new Bootstrap(); bootstrap.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel ch) throws Exception { ChannelPipeline pipeline = ch.pipeline(); // Encoder pipeline.addLast("encoder", new ObjectEncoder()); // Decoder pipeline.addLast("decoder", new ObjectDecoder(Integer.MAX_VALUE, ClassResolvers.cacheDisabled(null))); // Client Business Processing Class pipeline.addLast("handler", resultHandler); } }); ChannelFuture future = bootstrap.connect("127.0.0.1", 9999).sync(); future.channel().writeAndFlush(classInfo).sync(); future.channel().closeFuture().sync(); } finally { group.shutdownGracefully(); } return resultHandler.getResponse(); } }); } } package com.example.testdemo.rpc; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelInboundHandlerAdapter; import org.reflections.Reflections; import java.lang.reflect.Method; import java.util.Set; /** * Server-side Business Processing Class * */ public class InvokerHandler extends ChannelInboundHandlerAdapter { /** * Get the name of an implementation class under an interface * * @param classInfo * @return * @throws Exception */ private String getImplClassName(ClassInfo classInfo) throws Exception { // The package path where the server interface and implementation class are located String interfacePath = "com.example.testdemo.rpc"; int lastDot = classInfo.getClassName().lastIndexOf("."); String interfaceName = classInfo.getClassName().substring(lastDot); Class supperClass = Class.forName(interfacePath + interfaceName); Reflections reflection = new Reflections(interfacePath); // Get all the implementation classes under an interface Set<Class> implClassSet = reflection.getSubTypesOf(supperClass); if (implClassSet.size() == 0) { System.out.println("No implementation class found"); return null; } else if (implClassSet.size() > 1) { System.out.println("Find multiple implementation classes, which one is not explicitly used"); return null; } else { // Convert a set to an array Class[] classes = implClassSet.toArray(new Class[0]); // Get the name of the implementation class return classes[0].getName(); } } /** * Read data from client and implement method of class through reflection call * * @param ctx * @param msg * @throws Exception */ @Override public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception { ClassInfo classInfo = (ClassInfo)msg; Object clazz = Class.forName(getImplClassName(classInfo)).newInstance(); Method method = classInfo.getClass().getMethod(classInfo.getMethodName(), classInfo.getTypes()); // Method to implement a class through reflection calls Object result = method.invoke(clazz, classInfo.getObjects()); ctx.writeAndFlush(result); } } package com.example.testdemo.rpc; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.ChannelInboundHandlerAdapter; /** * Client Business Processing Class */ public class ResultHandler extends ChannelInboundHandlerAdapter { private Object response; public Object getResponse() { return response; } @Override public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception { response = msg; ctx.close(); } } package com.example.testdemo.rpc; import java.io.Serializable; public class ClassInfo implements Serializable { public String getClassName() { return className; } public void setClassName(String className) { this.className = className; } public String getMethodName() { return methodName; } public void setMethodName(String methodName) { this.methodName = methodName; } public Class<?>[] getTypes() { return types; } public void setTypes(Class<?>[] types) { this.types = types; } public Object[] getObjects() { return objects; } public void setObjects(Object[] objects) { this.objects = objects; } private static final long serialVersionUID = 1L; /** * Class name */ private String className; /** * Method Name */ private String methodName; /** * Parameter type */ private Class<?>[] types; /** * parameter list */ private Object[] objects; }