Message Queue Source Code Interpretation of NSQ Landing Disk

Keywords: Go github

The original address of this article

NSQ message queue implements message landing using FIFO queue.
The implementation is diskqueue, using the package This paper mainly introduces the implementation of diskqueue.
The code in this article comes from

Functional orientation

  • In NSQ, diskqueue is an instantiated Backend Queue that is used to store messages that cannot be stored in it. It is a more classical queue implementation and worth learning.
  • The functions implemented are a FIFO queue, which implements the following functions:

    • Support message insertion, clearing, deletion, closing operations
    • The length of the queue that can be returned (the distance between write and read offsets)
    • FIFO Queue with Read-Write Function

Implementation of diskqueue

diskqueue is the implementation of BackendQueue (the interface required for a queue), which is defined as follows:

type BackendQueue interface {
    Put([]byte) error      // Insert a message into the queue
    ReadChan() chan []byte // Returns a buffered chan
    Close() error          // Queue closure
    Delete() error         // Delete queues (data is still retained when actually implemented)
    Depth() int64          // Returns the amount of messages with read delays
    Empty() error          // Clean up the message (actually deleting all log files)

data structure

For 64 bits of fields that require atomic manipulation, you need to be at the top of struct. See Learning Summary Article 1 for reasons
The data structure defines the position of reading and writing files, some control variables of reading and writing files, and the channel of related operations.

// diskQueue implements a filesystem backed FIFO queue
type diskQueue struct {
    // 64bit atomic vars need to be first for proper alignment on 32bit platforms

    // run-time state (also persisted to disk)
    readPos      int64               // Reading position
    writePos     int64               // Writing position
    readFileNum  int64               // Number of Read Documents
    writeFileNum int64               // Number of Writing Documents
    depth        int64               // The distance between reading and writing files (used to identify the length of the queue)


    // instantiation time metadata
    name            string           // Identify queue name, prefix for landing file name 
    dataPath        string           // The path of the landing file
    maxBytesPerFile int64            // Maximum bytes per file
    minMsgSize      int32            // Minimum size of a single message
    maxMsgSize      int32            // Maximum Size of Single-Pick Messages
    syncEvery       int64            // How many times do you brush the dishes?
    syncTimeout     time.Duration    // How often do you brush the dishes at least?
    exitFlag        int32            // Exit sign
    needSync        bool             // If needSync is true, fsync is required to refresh metadata data data

    // keeps track of the position where we have read
    // (but not yet sent over readChan)
    nextReadPos     int64            // Next reading position
    nextReadFileNum int64            // number of files to be read next time

    readFile  *os.File               // Read fd
    writeFile *os.File               // Write fd
    reader    *bufio.Reader          // Read buffer
    writeBuf  bytes.Buffer           // Write buffer

    // exposed via ReadChan()
    readChan chan []byte             // Read channel

    // internal channels
    writeChan         chan []byte    // Write channel
    writeResponseChan chan error     // response after synchronization
    emptyChan         chan int       // Clear the channel of the file
    emptyResponseChan chan error     // channel after Synchronizing Clearing Files
    exitChan          chan int       // Exit channel
    exitSyncChan      chan int       // Exit command waiting for channel synchronously

    logf AppLogFunc                  // log handle

Initialize a queue

To initialize a queue, you need to define prefix names, data paths, maximum bytes per file, maximum and minimum message limits, as well as brush frequency and maximum brush time, and finally a log function.

func New(name string, dataPath string, maxBytesPerFile int64,
    minMsgSize int32, maxMsgSize int32,
    syncEvery int64, syncTimeout time.Duration, logf AppLogFunc) Interface {
    d := diskQueue{
        name:              name,
        dataPath:          dataPath,
        maxBytesPerFile:   maxBytesPerFile,
        minMsgSize:        minMsgSize,
        maxMsgSize:        maxMsgSize,
        readChan:          make(chan []byte),
        writeChan:         make(chan []byte),
        writeResponseChan: make(chan error),
        emptyChan:         make(chan int),
        emptyResponseChan: make(chan error),
        exitChan:          make(chan int),
        exitSyncChan:      make(chan int),
        syncEvery:         syncEvery,
        syncTimeout:       syncTimeout,
        logf:              logf,

    // no need to lock here, nothing else could possibly be touching this instance
    err := d.retrieveMetaData()
    if err != nil && !os.IsNotExist(err) {
        d.logf(ERROR, "DISKQUEUE(%s) failed to retrieveMetaData - %s",, err)

    go d.ioLoop()
    return &d

As you can see, chan without cache is used in all queues, and messages can only be blocked.

d.retrieveMetaData() recovers metadata from a file.

d.ioLoop() is the event processing logic of the queue, which will be explained in detail later.

Reading and Writing of Messages

file format

File name "name" +. diskqueue.% 06d. dat where name is topic, or topic + channel name.
Data is stored in binary mode and message size + body form.

Message read operation

  • If the readFile file descriptor is not initialized, you need to open the corresponding file first, offset the seek to the corresponding location, and initialize the reader buffer.
  • After initialization, first read the size of the file, 4 bytes, and then get the corresponding body data through the size of the file.
  • Change the corresponding offset. If the offset reaches the maximum value of the file, the corresponding file will be closed and the read file number + 1

Message writing operation

  • If the writeFile file descriptor is not initialized, you need to open the corresponding file first and offset the seek to the end of the file.
  • Verify that the size of the message meets the requirements
  • Write body size and body into buffer and land
  • depth +1,
  • If the file size is larger than the maximum size of each file, close the current file and write the file number + 1

Event Loop

ioLoop function, doing all the time processing operations, including:

  • Message read
  • Write operation
  • Empty queue data
  • Timely refresh events
func (d *diskQueue) ioLoop() {
    var dataRead []byte
    var err error
    var count int64
    var r chan []byte

    // Timer settings
    syncTicker := time.NewTicker(d.syncTimeout)

    for {
        // If the frequency of the brush is reached, mark the waiting brush
        if count == d.syncEvery {
            d.needSync = true

        if d.needSync {
            err = d.sync()
            if err != nil {
                d.logf(ERROR, "DISKQUEUE(%s) failed to sync - %s",, err)
            count = 0

        // If there is readable data, and the data currently read by chan has been read away, the next data is read.
        if (d.readFileNum < d.writeFileNum) || (d.readPos < d.writePos) {
            if d.nextReadPos == d.readPos {
                dataRead, err = d.readOne()
                if err != nil {
                    d.logf(ERROR, "DISKQUEUE(%s) reading at %d of %s - %s",
              , d.readPos, d.fileName(d.readFileNum), err)
            r = d.readChan
        } else {
            // If there is no readable data, set r to nil to prevent duplication of data Read data into readChan
            r = nil

        select {
        // the Go channel spec dictates that nil channel operations (read or write)
        // in a select are skipped, we set r to d.readChan only when there is data to read
        case r <- dataRead:
            // moveForward sets needSync flag if a file is removed
            // If the read chan is written successfully, the read offset is modified
        case <-d.emptyChan:
            // Clear all files and return empty results
            d.emptyResponseChan <- d.deleteAllFiles()
            count = 0
        case dataWrite := <-d.writeChan:
            // Write msg
            d.writeResponseChan <- d.writeOne(dataWrite)
        case <-syncTicker.C:
            // Modify needSync = true until brush time
            if count == 0 {
                // avoid sync when there's no activity
            d.needSync = true
        case <-d.exitChan:
            goto exit

    d.logf(INFO, "DISKQUEUE(%s): closing ... ioLoop",
    d.exitSyncChan <- 1

Points to be noted are as follows:

  1. The data is read out in advance, and when sent to readChan, the read offset is changed by the move Forward operation.
  2. queue's Put operation is non-operational and waits for the write to complete before returning the result
  3. Empty operations empty all data
  4. The data calls the FSync brush on a regular basis or at a set synchronous frequency.

Metadata metadata

metadata file format

File name: "name" +. diskqueue. meta. dat where name is topic, or topic + channel name.

The metadata data data data contains five fields, which are as follows:


metadata effect

When the service is closed, metadata data data is saved in the file. When the service starts again, the relevant data is restored from the file to memory.

Learning summary

Memory Alignment and Atomic Operation

// 64bit atomic vars need to be first for proper alignment on 32bit platforms
  • Phenomenon nsq, when defining struct, has many similar annotations
  • The reason is in the golang source sync/atomic/doc.go

    // On ARM, x86-32, and 32-bit MIPS,
    // it is the caller's responsibility to arrange for 64-bit
    // alignment of 64-bit words accessed atomically. The first word in a
    // variable or in an allocated struct, array, or slice can be relied upon to be
    // 64-bit aligned.
  • Explain that in arm, 32 x86 system, and 32-bit MIPS instruction set, the caller needs to ensure that 64-bit memory alignment (rather than 32-bit alignment) is performed for 64-bit variables atomically. Putting 64-bit variables at the front of struct, array, slice can ensure 64-bit alignment.
  • Conclusion Variables with 64 bit atomic operation are defined at the front of struct, which can align 64 bits of variables and ensure the correct execution of programs in 32 bits system.

Use of Object Pool

  • In buffer_pool.go file, the object pool of bytes.Buffer is implemented simply, which reduces gc pressure.
  • With scenarios, high frequency object initialization and memory allocation are required. sync.Pool object pool can be used to reduce gc pressure.

How to actively refresh the data in the operating system cache to the hard disk?

  • Fsync function (after the write function, you need fsync to ensure that the data drops)

Posted by bukuru on Tue, 10 Sep 2019 20:38:40 -0700