Java Network Programming--ByteBuf in Netty

Keywords: Programming Netty less JDK Java

Because the ByteBuffer provided in JDK can not be dynamically expanded and the API is complex to use, Netty provides ByteBuffer. The API of Bytebuf is more convenient and can be expanded dynamically. It provides a variety of implementations of Bytebuf and an efficient zero-copy mechanism.

Operation of ByteBuf

ByteBuf has three important attributes: capacity capacity, readerIndex read location, and writerIndex write location. Two variable pointers, readerIndex and weiterIndex, are provided to support sequential read and write operations.

The following figure shows how a buffer is divided into three regions by two pointers:

Code example:

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;

import java.util.Arrays;

public class ByteBufDemo {
  public static void main(String[] args) {
    // 1. Create a non-pooled ByteBuf of 10 bytes in size
    ByteBuf buf = Unpooled.buffer(10);
    System.out.println("Original ByteBuf For:" + buf.toString());
    System.out.println("1.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 2. Write a paragraph
    byte[] bytes = {1, 2, 3, 4, 5};
    buf.writeBytes(bytes);
    System.out.println("Write in bytes For:" + Arrays.toString(bytes));
    System.out.println("After writing a paragraph ByteBuf For:" + buf.toString());
    System.out.println("2.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 3. Read a paragraph
    byte b1 = buf.readByte();
    byte b2 = buf.readByte();
    System.out.println("Read out bytes For:" + Arrays.toString(new byte[] {b1, b2}));
    System.out.println("After reading a paragraph ByteBuf For:" + buf.toString());
    System.out.println("3.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 4. Discard what you read
    buf.discardReadBytes();
    System.out.println("After discarding the read content ByteBuf For:" + buf.toString());
    System.out.println("4.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 5. Clear the Read-Write Pointer
    buf.clear();
    System.out.println("After clearing the read and write pointer ByteBuf For:" + buf.toString());
    System.out.println("5.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 6. Write a paragraph again, less than the first one.
    byte[] bytes2 = {1, 2, 3};
    buf.writeBytes(bytes2);
    System.out.println("Write in bytes For:" + Arrays.toString(bytes2));
    System.out.println("After writing a paragraph ByteBuf For:" + buf.toString());
    System.out.println("6.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 7. Zero ByteBuf
    buf.setZero(0, buf.capacity());
    System.out.println("After clearing ByteBuf For:" + buf.toString());
    System.out.println("7.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");

    // 8. Write a section of content that exceeds capacity again
    byte[] bytes3 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11};
    buf.writeBytes(bytes3);
    System.out.println("Write in bytes For:" + Arrays.toString(bytes));
    System.out.println("After writing a paragraph ByteBuf For:" + buf.toString());
    System.out.println("8.ByteBuf The contents are as follows:" + Arrays.toString(buf.array()) + "\n");
  }
}

ButeBuf dynamic expansion

capacity default: 256 bytes, maximum: Integer.MAX_VALUE (2G)

When the writeXXX method is called, it is checked by the AbstractByteBuf.ensureWritable0() method Capacity calculation method: AbstractByteBufAllocator. CalulateNewCapacity

According to the minimum value requirement of capacity, there are two sets of calculation methods: Less than 4 megabytes: Start with 64 bytes, doubling each time until the calculated new Capacity meets the minimum new capacity requirements Example: Current size 256, written 250, continue to write 10 bytes of data, the minimum required capacity is 261, then the new capacity is 64x2x2x2=512

Over 4 Megabytes: New Capacity = New Capacity Minimum Requirements / 4 Megabytes x 4 Megabytes + 4 Megabytes Example: The current size is 3 megabytes, 3 megabytes have been written, and 2 megabytes continue to be written. The minimum required capacity is 5 megabytes, and the new capacity is 8 megabytes (not exceeding the maximum).

Source of 4M: A fixed threshold AbstractByteBufAllocator.CALCULATE_THRESHOLD

Implementation of ByteBuf

In use, all applications are made through ByteBuf Allocator allocator with memory management functions.

Pooled ByteBuf object, memory reuse

Pooled ThreadCache: A thread variable maintained by the Pooled ByteBufAllocator instance Multiple classifications of the MemoryRegionCache array are used as memory caches. MemoryRegionCache has a linked list inside and a Chuck in the queue. Memory references are maintained in Pool Chuck, and memory reuse is done by pointing buf's memory to chuck's memory. Pooled ByteBuf Allocator.ioBuffer's operation process is sorted out:

Zero Copy Mechanism

The zero copy mechanism of Netty is an application layer implementation, which is not much related to the underlying JVM and the memory mechanism of the operating system.

Composite ByteBuf, which combines multiple ByteBufs into a logical ByteBuf, avoids copying between ByteBufs

wrapedBuffer() method that wraps byte [] arrays into ByteBuf objects

The slice() method cuts a ByteBuf object into multiple ByteBuf objects