The original address of this article https://blog.lpflpf.cn/passages/nsqd-study-2/
NSQ message queue implements message landing using FIFO queue.
The implementation is diskqueue, using the package github.com/nsqio/go-diskqueue. This paper mainly introduces the implementation of diskqueue.
The code in this article comes from github.com/nsqio/go-diskqueue
Functional orientation
- In NSQ, diskqueue is an instantiated Backend Queue that is used to store messages that cannot be stored in it. It is a more classical queue implementation and worth learning.
-
The functions implemented are a FIFO queue, which implements the following functions:
- Support message insertion, clearing, deletion, closing operations
- The length of the queue that can be returned (the distance between write and read offsets)
- FIFO Queue with Read-Write Function
Implementation of diskqueue
diskqueue is the implementation of BackendQueue (the interface required for a queue), which is defined as follows:
type BackendQueue interface { Put([]byte) error // Insert a message into the queue ReadChan() chan []byte // Returns a buffered chan Close() error // Queue closure Delete() error // Delete queues (data is still retained when actually implemented) Depth() int64 // Returns the amount of messages with read delays Empty() error // Clean up the message (actually deleting all log files) }
data structure
For 64 bits of fields that require atomic manipulation, you need to be at the top of struct. See Learning Summary Article 1 for reasons
The data structure defines the position of reading and writing files, some control variables of reading and writing files, and the channel of related operations.
// diskQueue implements a filesystem backed FIFO queue type diskQueue struct { // 64bit atomic vars need to be first for proper alignment on 32bit platforms // run-time state (also persisted to disk) readPos int64 // Reading position writePos int64 // Writing position readFileNum int64 // Number of Read Documents writeFileNum int64 // Number of Writing Documents depth int64 // The distance between reading and writing files (used to identify the length of the queue) sync.RWMutex // instantiation time metadata name string // Identify queue name, prefix for landing file name dataPath string // The path of the landing file maxBytesPerFile int64 // Maximum bytes per file minMsgSize int32 // Minimum size of a single message maxMsgSize int32 // Maximum Size of Single-Pick Messages syncEvery int64 // How many times do you brush the dishes? syncTimeout time.Duration // How often do you brush the dishes at least? exitFlag int32 // Exit sign needSync bool // If needSync is true, fsync is required to refresh metadata data data // keeps track of the position where we have read // (but not yet sent over readChan) nextReadPos int64 // Next reading position nextReadFileNum int64 // number of files to be read next time readFile *os.File // Read fd writeFile *os.File // Write fd reader *bufio.Reader // Read buffer writeBuf bytes.Buffer // Write buffer // exposed via ReadChan() readChan chan []byte // Read channel // internal channels writeChan chan []byte // Write channel writeResponseChan chan error // response after synchronization emptyChan chan int // Clear the channel of the file emptyResponseChan chan error // channel after Synchronizing Clearing Files exitChan chan int // Exit channel exitSyncChan chan int // Exit command waiting for channel synchronously logf AppLogFunc // log handle }
Initialize a queue
To initialize a queue, you need to define prefix names, data paths, maximum bytes per file, maximum and minimum message limits, as well as brush frequency and maximum brush time, and finally a log function.
func New(name string, dataPath string, maxBytesPerFile int64, minMsgSize int32, maxMsgSize int32, syncEvery int64, syncTimeout time.Duration, logf AppLogFunc) Interface { d := diskQueue{ name: name, dataPath: dataPath, maxBytesPerFile: maxBytesPerFile, minMsgSize: minMsgSize, maxMsgSize: maxMsgSize, readChan: make(chan []byte), writeChan: make(chan []byte), writeResponseChan: make(chan error), emptyChan: make(chan int), emptyResponseChan: make(chan error), exitChan: make(chan int), exitSyncChan: make(chan int), syncEvery: syncEvery, syncTimeout: syncTimeout, logf: logf, } // no need to lock here, nothing else could possibly be touching this instance err := d.retrieveMetaData() if err != nil && !os.IsNotExist(err) { d.logf(ERROR, "DISKQUEUE(%s) failed to retrieveMetaData - %s", d.name, err) } go d.ioLoop() return &d }
As you can see, chan without cache is used in all queues, and messages can only be blocked.
d.retrieveMetaData() recovers metadata from a file.
d.ioLoop() is the event processing logic of the queue, which will be explained in detail later.
Reading and Writing of Messages
file format
File name "name" +. diskqueue.% 06d. dat where name is topic, or topic + channel name.
Data is stored in binary mode and message size + body form.
Message read operation
- If the readFile file descriptor is not initialized, you need to open the corresponding file first, offset the seek to the corresponding location, and initialize the reader buffer.
- After initialization, first read the size of the file, 4 bytes, and then get the corresponding body data through the size of the file.
- Change the corresponding offset. If the offset reaches the maximum value of the file, the corresponding file will be closed and the read file number + 1
Message writing operation
- If the writeFile file descriptor is not initialized, you need to open the corresponding file first and offset the seek to the end of the file.
- Verify that the size of the message meets the requirements
- Write body size and body into buffer and land
- depth +1，
- If the file size is larger than the maximum size of each file, close the current file and write the file number + 1
Event Loop
ioLoop function, doing all the time processing operations, including:
- Message read
- Write operation
- Empty queue data
- Timely refresh events
func (d *diskQueue) ioLoop() { var dataRead []byte var err error var count int64 var r chan []byte // Timer settings syncTicker := time.NewTicker(d.syncTimeout) for { // If the frequency of the brush is reached, mark the waiting brush if count == d.syncEvery { d.needSync = true } if d.needSync { err = d.sync() if err != nil { d.logf(ERROR, "DISKQUEUE(%s) failed to sync - %s", d.name, err) } count = 0 } // If there is readable data, and the data currently read by chan has been read away, the next data is read. if (d.readFileNum < d.writeFileNum) || (d.readPos < d.writePos) { if d.nextReadPos == d.readPos { dataRead, err = d.readOne() if err != nil { d.logf(ERROR, "DISKQUEUE(%s) reading at %d of %s - %s", d.name, d.readPos, d.fileName(d.readFileNum), err) d.handleReadError() continue } } r = d.readChan } else { // If there is no readable data, set r to nil to prevent duplication of data Read data into readChan r = nil } select { // the Go channel spec dictates that nil channel operations (read or write) // in a select are skipped, we set r to d.readChan only when there is data to read case r <- dataRead: count++ // moveForward sets needSync flag if a file is removed // If the read chan is written successfully, the read offset is modified d.moveForward() case <-d.emptyChan: // Clear all files and return empty results d.emptyResponseChan <- d.deleteAllFiles() count = 0 case dataWrite := <-d.writeChan: // Write msg count++ d.writeResponseChan <- d.writeOne(dataWrite) case <-syncTicker.C: // Modify needSync = true until brush time if count == 0 { // avoid sync when there's no activity continue } d.needSync = true case <-d.exitChan: goto exit } } exit: d.logf(INFO, "DISKQUEUE(%s): closing ... ioLoop", d.name) syncTicker.Stop() d.exitSyncChan <- 1 }
Points to be noted are as follows:
- The data is read out in advance, and when sent to readChan, the read offset is changed by the move Forward operation.
- queue's Put operation is non-operational and waits for the write to complete before returning the result
- Empty operations empty all data
- The data calls the FSync brush on a regular basis or at a set synchronous frequency.
Metadata metadata
metadata file format
File name: "name" +. diskqueue. meta. dat where name is topic, or topic + channel name.
The metadata data data data contains five fields, which are as follows:
depth\nreadFileNum,readPos\nwriteFileNum,writePos
metadata effect
When the service is closed, metadata data data is saved in the file. When the service starts again, the relevant data is restored from the file to memory.
Learning summary
Memory Alignment and Atomic Operation
// 64bit atomic vars need to be first for proper alignment on 32bit platforms
- Phenomenon nsq, when defining struct, has many similar annotations
-
The reason is in the golang source sync/atomic/doc.go
// On ARM, x86-32, and 32-bit MIPS, // it is the caller's responsibility to arrange for 64-bit // alignment of 64-bit words accessed atomically. The first word in a // variable or in an allocated struct, array, or slice can be relied upon to be // 64-bit aligned. - Explain that in arm, 32 x86 system, and 32-bit MIPS instruction set, the caller needs to ensure that 64-bit memory alignment (rather than 32-bit alignment) is performed for 64-bit variables atomically. Putting 64-bit variables at the front of struct, array, slice can ensure 64-bit alignment.
- Conclusion Variables with 64 bit atomic operation are defined at the front of struct, which can align 64 bits of variables and ensure the correct execution of programs in 32 bits system.
Use of Object Pool
- In buffer_pool.go file, the object pool of bytes.Buffer is implemented simply, which reduces gc pressure.
- With scenarios, high frequency object initialization and memory allocation are required. sync.Pool object pool can be used to reduce gc pressure.
How to actively refresh the data in the operating system cache to the hard disk?
- Fsync function (after the write function, you need fsync to ensure that the data drops)