RocketMQ Source Parsing: Message Extraction & Consumption (Part I)

Keywords: JSON less github

> Original address: RocketMQ Source Parsing: Message Extraction & Consumption (Part I)
> RocketMQ with annotated address: YunaiV/incubator-rocketmq
> (vii) This series is updated every 1-2 weeks. You are welcome to subscribe, follow and collect GitHub.

1. Overview

This chapter mainly analyses the source code involved in the consumption logic. Because of its long length, it is divided into two parts:

Part I: Broker related source code.
Next: Consumer related source code.

This article is the first part.

ok, first look at the first picture about the logic of consumption:

Look at the sequence diagram of the streamlined consumption logic (the actual situation will be slightly different):

2. ConsumeQueue structure

The relationships among ConsumeQueue, MappedFileQueue and MappedFile are as follows:

> ConsumeQueue : MappedFileQueue : MappedFile = 1 : 1 : N.

The response to the system file is as follows:

Yunai-MacdeMacBook-Pro-2:consumequeue yunai$ pwd
/Users/yunai/store/consumequeue
Yunai-MacdeMacBook-Pro-2:consumequeue yunai$ cd TopicRead3/
Yunai-MacdeMacBook-Pro-2:TopicRead3 yunai$ ls -ls
total 0
0 drwxr-xr-x  3 yunai  staff  102  4 27 21:52 0
0 drwxr-xr-x  3 yunai  staff  102  4 27 21:55 1
0 drwxr-xr-x  3 yunai  staff  102  4 27 21:55 2
0 drwxr-xr-x  3 yunai  staff  102  4 27 21:55 3
Yunai-MacdeMacBook-Pro-2:TopicRead3 yunai$ cd 0/
Yunai-MacdeMacBook-Pro-2:0 yunai$ ls -ls
total 11720
11720 -rw-r--r--  1 yunai  staff  6000000  4 27 21:55 00000000000000000000

ConsumeQueue, MappedFileQueue and MappedFile are defined as follows:

MappedFile: 000 000 000 000 000 000 000 000 000 000 etc.
MappedFileQueue: The folder where MappedFile is located. MappedFile is encapsulated into a file queue to provide unlimited file capacity for the upper layer.
- Each MappedFile has a uniform file size.
- File naming: fileName[n] = fileName[n - 1] + mappedFileSize. The default in Consume Queue is 6000000B.
ConsumeQueue: Encapsulation for MappedFileQueue.
- Store : ConsumeQueue = ConcurrentHashMap<String/* topic */, ConcurrentHashMap<Integer/* queueId */, ConsumeQueue>>.

ConsumeQueue content stored in MappedFile must be 20B in size (ConsumeQueue.CQ_STORE_UNIT_SIZE). There are two types of content:

MESSAGE_POSITION_INFO: Message location information.
BLANK: Blank space in front of the file. When a historical Message is deleted, it is necessary to use BLANK to occupy the deleted Message.

MESSAGE_POSITION_INFO in ConsumeQueue storage structure:

Which Number	field	Explain	data type	Number of bytes
1	offset	Message CommitLog Storage Location	Long	8
2	size	Message length	Int	4
3	tagsCode	Message tagsCode	Long	8

BLANK in ConsumeQueue Storage Architecture:

Which Number	Explain	data type	Number of bytes
1	0	Long	8
2	Integer.MAX_VALUE	Int	4
3	0	Long	8

3. ConsumeQueue Storage

There are two main components:

ReputMessageService : write ConsumeQueue.
FlushConsumeQueueService : flush ConsumeQueue.

ReputMessageService

  1: class ReputMessageService extends ServiceThread {
  2: 
  3:     /**
  4:      * CommitLog physical location to start replaying messages
  5:      */
  6:     private volatile long reputFromOffset = 0;
  7: 
  8:     public long getReputFromOffset() {
  9:         return reputFromOffset;
 10:     }
 11: 
 12:     public void setReputFromOffset(long reputFromOffset) {
 13:         this.reputFromOffset = reputFromOffset;
 14:     }
 15: 
 16:     @Override
 17:     public void shutdown() {
 18:         for (int i = 0; i < 50 && this.isCommitLogAvailable(); i++) {
 19:             try {
 20:                 Thread.sleep(100);
 21:             } catch (InterruptedException ignored) {
 22:             }
 23:         }
 24: 
 25:         if (this.isCommitLogAvailable()) {
 26:             log.warn("shutdown ReputMessageService, but commitlog have not finish to be dispatched, CL: {} reputFromOffset: {}",
 27:                 DefaultMessageStore.this.commitLog.getMaxOffset(), this.reputFromOffset);
 28:         }
 29: 
 30:         super.shutdown();
 31:     }
 32: 
 33:     /**
 34:      * The remaining number of bytes of messages that need to be replayed
 35:      *
 36:      * @return Number of bytes
 37:      */
 38:     public long behind() {
 39:         return DefaultMessageStore.this.commitLog.getMaxOffset() - this.reputFromOffset;
 40:     }
 41: 
 42:     /**
 43:      * Whether commitLog needs to replay messages
 44:      *
 45:      * @return Whether or not?
 46:      */
 47:     private boolean isCommitLogAvailable() {
 48:         return this.reputFromOffset < DefaultMessageStore.this.commitLog.getMaxOffset();
 49:     }
 50: 
 51:     private void doReput() {
 52:         for (boolean doNext = true; this.isCommitLogAvailable() && doNext; ) {
 53: 
 54:             // TODO Question: What is this?
 55:             if (DefaultMessageStore.this.getMessageStoreConfig().isDuplicationEnable() //
 56:                 && this.reputFromOffset >= DefaultMessageStore.this.getConfirmOffset()) {
 57:                 break;
 58:             }
 59: 
 60:             // Get MappedByteBuffer for MappeFile corresponding to commitLog starting from reputFromOffset
 61:             SelectMappedBufferResult result = DefaultMessageStore.this.commitLog.getData(reputFromOffset);
 62:             if (result != null) {
 63:                 try {
 64:                     this.reputFromOffset = result.getStartOffset();
 65: 
 66:                     // Traversing MappedByteBuffer
 67:                     for (int readSize = 0; readSize < result.getSize() && doNext; ) {
 68:                         // Generate playback message playback scheduling request
 69:                         DispatchRequest dispatchRequest = DefaultMessageStore.this.commitLog.checkMessageAndReturnSize(result.getByteBuffer(), false, false);
 70:                         int size = dispatchRequest.getMsgSize(); // Message length
 71:                         // Processing based on the result of the request
 72:                         if (dispatchRequest.isSuccess()) { // Read successfully
 73:                             if (size > 0) { // Read Message
 74:                                 DefaultMessageStore.this.doDispatch(dispatchRequest);
 75:                                 // Notification of new information
 76:                                 if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
 77:                                     && DefaultMessageStore.this.brokerConfig.isLongPollingEnable()) {
 78:                                     DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
 79:                                         dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
 80:                                         dispatchRequest.getTagsCode());
 81:                                 }
 82:                                 // FIXED BUG By shijia
 83:                                 this.reputFromOffset += size;
 84:                                 readSize += size;
 85:                                 // Statistics
 86:                                 if (DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole() == BrokerRole.SLAVE) {
 87:                                     DefaultMessageStore.this.storeStatsService
 88:                                         .getSinglePutMessageTopicTimesTotal(dispatchRequest.getTopic()).incrementAndGet();
 89:                                     DefaultMessageStore.this.storeStatsService
 90:                                         .getSinglePutMessageTopicSizeTotal(dispatchRequest.getTopic())
 91:                                         .addAndGet(dispatchRequest.getMsgSize());
 92:                                 }
 93:                             } else if (size == 0) { // Read to the end of MappedFile file
 94:                                 this.reputFromOffset = DefaultMessageStore.this.commitLog.rollNextFile(this.reputFromOffset);
 95:                                 readSize = result.getSize();
 96:                             }
 97:                         } else if (!dispatchRequest.isSuccess()) { // read failure
 98:                             if (size > 0) { // Read Message but not Message
 99:                                 log.error("[BUG]read total count not equals msg total size. reputFromOffset={}", reputFromOffset);
100:                                 this.reputFromOffset += size;
101:                             } else { // Read Blank but not Blank
102:                                 doNext = false;
103:                                 if (DefaultMessageStore.this.brokerConfig.getBrokerId() == MixAll.MASTER_ID) {
104:                                     log.error("[BUG]the master dispatch message to consume queue error, COMMITLOG OFFSET: {}",
105:                                         this.reputFromOffset);
106: 
107:                                     this.reputFromOffset += result.getSize() - readSize;
108:                                 }
109:                             }
110:                         }
111:                     }
112:                 } finally {
113:                     result.release();
114:                 }
115:             } else {
116:                 doNext = false;
117:             }
118:         }
119:     }
120: 
121:     @Override
122:     public void run() {
123:         DefaultMessageStore.log.info(this.getServiceName() + " service started");
124: 
125:         while (!this.isStopped()) {
126:             try {
127:                 Thread.sleep(1);
128:                 this.doReput();
129:             } catch (Exception e) {
130:                 DefaultMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
131:             }
132:         }
133: 
134:         DefaultMessageStore.log.info(this.getServiceName() + " service end");
135:     }
136: 
137:     @Override
138:     public String getServiceName() {
139:         return ReputMessageService.class.getSimpleName();
140:     }
141: 
142: }

Description: Replay message thread service.
- The service continuously generates message location information to Consume Queue
- The service continuously generates message indexes to index files (IndexFile)
- Line 61: Get MappedByteBuffer for MappedFile corresponding to CommitLog starting from reputFromOffset.
- Line 67: Traverse MappedByteBuffer.
- Line 69: Generate Dispatch Request for replay messages. The request mainly contains the basic information of a message or the end of a file (BLANK).
- Lines 72 to 96: Requests are valid requests and are processed logically.
  - Lines 75 to 81: When Broker is the primary node & Broker opens a long poll to notify the consumer queue of new messages. NotifyMessageArriving Listener calls the PullRequestHoldService notifyMessageArriving (...) method. For detailed analysis, see: PullRequestHoldService
- Lines 73 to 92: The request corresponds to Message, which is scheduled to generate the corresponding contents of ConsumeQueue and IndexFile. For detailed analysis, see:
- Lines 93 to 96: The request corresponds to Blank, the end of the file, and the jump points to the next MappedFile.
- Lines 97 to 110: The request is invalid. In this case, it is basically a BUG.
Lines 127 to 128: Replay logic is performed per MS loop.
Lines 18 to 30: When shutdown, sleep(100) many times until CommitLog is put back to its latest location. Well, if the playback is not finished, the warning log will be output.

DefaultMessageStore#doDispatch(...)

  1: /**
  2:  * Execute scheduling requests
  3:  * 1. Non-transactional message or transaction submission message establishes message location information to ConsumeQueue
  4:  * 2. Create index information to IndexFile
  5:  *
  6:  * @param req Scheduling requests
  7:  */
  8: public void doDispatch(DispatchRequest req) {
  9:     // Non-transactional message or transaction submission message establishes message location information to ConsumeQueue
 10:     final int tranType = MessageSysFlag.getTransactionValue(req.getSysFlag());
 11:     switch (tranType) {
 12:         case MessageSysFlag.TRANSACTION_NOT_TYPE:
 13:         case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
 14:             DefaultMessageStore.this.putMessagePositionInfo(req.getTopic(), req.getQueueId(), req.getCommitLogOffset(), req.getMsgSize(),
 15:                 req.getTagsCode(), req.getStoreTimestamp(), req.getConsumeQueueOffset());
 16:             break;
 17:         case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
 18:         case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
 19:             break;
 20:     }
 21:     // Create index information to IndexFile
 22:     if (DefaultMessageStore.this.getMessageStoreConfig().isMessageIndexEnable()) {
 23:         DefaultMessageStore.this.indexService.buildIndex(req);
 24:     }
 25: }
 26: 
 27: /**
 28:  * Establish message location information to ConsumeQueue
 29:  *
 30:  * @param topic theme
 31:  * @param queueId Queue number
 32:  * @param offset commitLog Storage location
 33:  * @param size Message length
 34:  * @param tagsCode Message tagsCode
 35:  * @param storeTimestamp Storage time
 36:  * @param logicOffset Queue position
 37:  */
 38: public void putMessagePositionInfo(String topic, int queueId, long offset, int size, long tagsCode, long storeTimestamp,
 39:     long logicOffset) {
 40:     ConsumeQueue cq = this.findConsumeQueue(topic, queueId);
 41:     cq.putMessagePositionInfoWrapper(offset, size, tagsCode, storeTimestamp, logicOffset);
 42: }

ConsumeQueue#putMessagePositionInfoWrapper(...)

  1: /**
  2:  * Add location information encapsulation
  3:  *
  4:  * @param offset commitLog Storage location
  5:  * @param size Message length
  6:  * @param tagsCode Message tagsCode
  7:  * @param storeTimestamp Message Storage Time
  8:  * @param logicOffset Queue position
  9:  */
 10: public void putMessagePositionInfoWrapper(long offset, int size, long tagsCode, long storeTimestamp,
 11:     long logicOffset) {
 12:     final int maxRetries = 30;
 13:     boolean canWrite = this.defaultMessageStore.getRunningFlags().isWriteable();
 14:     // Repeated iterations until success
 15:     for (int i = 0; i < maxRetries && canWrite; i++) {
 16:         // Call add location information
 17:         boolean result = this.putMessagePositionInfo(offset, size, tagsCode, logicOffset);
 18:         if (result) {
 19:             // Successful addition, using message storage time as storage check point.
 20:             this.defaultMessageStore.getStoreCheckpoint().setLogicsMsgTimestamp(storeTimestamp);
 21:             return;
 22:         } else {
 23:             // XXX: warn and notify me
 24:             log.warn("[BUG]put commit log position info to " + topic + ":" + queueId + " " + offset
 25:                 + " failed, retry " + i + " times");
 26: 
 27:             try {
 28:                 Thread.sleep(1000);
 29:             } catch (InterruptedException e) {
 30:                 log.warn("", e);
 31:             }
 32:         }
 33:     }
 34: 
 35:     // XXX: warn and notify me setting exception not writable
 36:     log.error("[BUG]consume queue can not write, {} {}", this.topic, this.queueId);
 37:     this.defaultMessageStore.getRunningFlags().makeLogicsQueueError();
 38: }
 39: 
 40: /**
 41:  * Add location information and return whether the addition was successful
 42:  *
 43:  * @param offset commitLog Storage location
 44:  * @param size Message length
 45:  * @param tagsCode Message tagsCode
 46:  * @param cqOffset Queue position
 47:  * @return Success or not
 48:  */
 49: private boolean putMessagePositionInfo(final long offset, final int size, final long tagsCode,
 50:     final long cqOffset) {
 51:     // If replayed, return directly to success
 52:     if (offset <= this.maxPhysicOffset) {
 53:         return true;
 54:     }
 55:     // Write location information to byteBuffer
 56:     this.byteBufferIndex.flip();
 57:     this.byteBufferIndex.limit(CQ_STORE_UNIT_SIZE);
 58:     this.byteBufferIndex.putLong(offset);
 59:     this.byteBufferIndex.putInt(size);
 60:     this.byteBufferIndex.putLong(tagsCode);
 61:     // Calculate the consumeQueue storage location and get the corresponding MappedFile
 62:     final long expectLogicOffset = cqOffset * CQ_STORE_UNIT_SIZE;
 63:     MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile(expectLogicOffset);
 64:     if (mappedFile != null) {
 65:         // When ConsumeQueue's first MappedFile & Queue location is not the first & MappedFile does not write content, fill in the preemptive blank placement
 66:         if (mappedFile.isFirstCreateInQueue() && cqOffset != 0 && mappedFile.getWrotePosition() == 0) { // TODO Question: Why this operation? What can be imagined at present is that some old messages have not been sent for a long time, and suddenly sent, at this time just satisfied.
 67:             this.minLogicOffset = expectLogicOffset;
 68:             this.mappedFileQueue.setFlushedWhere(expectLogicOffset);
 69:             this.mappedFileQueue.setCommittedWhere(expectLogicOffset);
 70:             this.fillPreBlank(mappedFile, expectLogicOffset);
 71:             log.info("fill pre blank space " + mappedFile.getFileName() + " " + expectLogicOffset + " "
 72:                 + mappedFile.getWrotePosition());
 73:         }
 74:         // Verify that the consumeQueue storage location is legitimate. If TODO is not legal, will it be a problem to continue writing?
 75:         if (cqOffset != 0) {
 76:             long currentLogicOffset = mappedFile.getWrotePosition() + mappedFile.getFileFromOffset();
 77:             if (expectLogicOffset != currentLogicOffset) {
 78:                 LOG_ERROR.warn(
 79:                     "[BUG]logic queue order maybe wrong, expectLogicOffset: {} currentLogicOffset: {} Topic: {} QID: {} Diff: {}",
 80:                     expectLogicOffset,
 81:                     currentLogicOffset,
 82:                     this.topic,
 83:                     this.queueId,
 84:                     expectLogicOffset - currentLogicOffset
 85:                 );
 86:             }
 87:         }
 88:         // Set the commitLog replay message to the ConsumeQueue location.
 89:         this.maxPhysicOffset = offset;
 90:         // Insert mappedFile
 91:         return mappedFile.appendMessage(this.byteBufferIndex.array());
 92:     }
 93:     return false;
 94: }
 95: 
 96: /**
 97:  * Fill in preemptive blank space
 98:  *
 99:  * @param mappedFile MappedFile
100:  * @param untilWhere consumeQueue Storage location
101:  */
102: private void fillPreBlank(final MappedFile mappedFile, final long untilWhere) {
103:     // Write preemptive space to byteBuffer
104:     ByteBuffer byteBuffer = ByteBuffer.allocate(CQ_STORE_UNIT_SIZE);
105:     byteBuffer.putLong(0L);
106:     byteBuffer.putInt(Integer.MAX_VALUE);
107:     byteBuffer.putLong(0L);
108:     // Cyclic filling
109:     int until = (int) (untilWhere % this.mappedFileQueue.getMappedFileSize());
110:     for (int i = 0; i < until; i += CQ_STORE_UNIT_SIZE) {
111:         mappedFile.appendMessage(byteBuffer.array());
112:     }
113: }

# PuputMessagePositionInfoWrapper (...) Description: To add location information to the encapsulation of ConsumeQueue, you actually need to call the # putMessagePositionInfo(...) method.
- Line 13: Determine whether ConsumeQueue is allowed to write. Writing is not allowed when a Bug occurs.
- Line 17: Call the # putMessagePositionInfo(...) method to add location information.
- Lines 18 to 21: Successful addition, using message storage time as storage checkpoint. For a detailed analysis of StoreCheckpoint, see: Store Initialization and Closing.
- Lines 22 to 32: Failure to add is currently considered BUG.
- Lines 35 to 37: When a write fails, the tag ConsumeQueue writes an exception, and no further write is allowed.
# PuputMessagePositionInfo (...) Describes: Add location information to ConsumeQueue and return whether the addition was successful.
- Lines 51 to 54: If offset (storage location) is less than or equal to maxPhysicOffset(CommitLog message is replayed to the largest CommitLog storage location in ConsumeQueue), it indicates that it has been replayed. At this time, no repetition of writing is required and the successful writing is returned directly.
- Lines 55 to 60: Write location information to byteBuffer.
- Lines 62 to 63: Compute the ConsumeQueue storage location and get the corresponding MappedFile.
- Lines 65 to 73: When MappedFile is the first current file of ConsumeQueue & MappedFile does not write content & the replay message queue location is greater than 0, the MappedFile is required to fill the pre-BLANK.
  - It's doubtful what scenario will be needed. The reason for this speculation is that a Topic has no message for a long time, and suddenly sends it N days later. The corresponding historical messages and consumption queue data of Topic have been cleaned up. The newly generated MappedFile needs to be pre-occupied.
- Lines 74 to 87: Verify that the ConsumeQueue storage location is legitimate and that the logs are exported illegally.
  - This piece is more questionable, if the calculated storage location is not valid, do not return to add failure, continue to add location information, will there be a problem???
- Line 89: Set CommitLog's maximum position for replaying messages to ConsumeQueue.
- Line 91: Insert the message location to MappedFile.

FlushConsumeQueueService

  1: class FlushConsumeQueueService extends ServiceThread {
  2:     private static final int RETRY_TIMES_OVER = 3;
  3:     /**
  4:      * Last flush timestamp
  5:      */
  6:     private long lastFlushTimestamp = 0;
  7: 
  8:     private void doFlush(int retryTimes) {
  9:         int flushConsumeQueueLeastPages = DefaultMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueLeastPages();
 10: 
 11:         // When retryTimes = RETRY_TIMES_OVER, force flush. Mainly used for shutdown.
 12:         if (retryTimes == RETRY_TIMES_OVER) {
 13:             flushConsumeQueueLeastPages = 0;
 14:         }
 15:         // When the time satisfies flush Consume Queue Thorough Interval, flush is performed even if the number of writes is less than that of flush Consume Queue Least Pages.
 16:         long logicsMsgTimestamp = 0;
 17:         int flushConsumeQueueThoroughInterval = DefaultMessageStore.this.getMessageStoreConfig().getFlushConsumeQueueThoroughInterval();
 18:         long currentTimeMillis = System.currentTimeMillis();
 19:         if (currentTimeMillis >= (this.lastFlushTimestamp + flushConsumeQueueThoroughInterval)) {
 20:             this.lastFlushTimestamp = currentTimeMillis;
 21:             flushConsumeQueueLeastPages = 0;
 22:             logicsMsgTimestamp = DefaultMessageStore.this.getStoreCheckpoint().getLogicsMsgTimestamp();
 23:         }
 24:         // flush consumption queue
 25:         ConcurrentHashMap<String, ConcurrentHashMap<Integer, ConsumeQueue>> tables = DefaultMessageStore.this.consumeQueueTable;
 26:         for (ConcurrentHashMap<Integer, ConsumeQueue> maps : tables.values()) {
 27:             for (ConsumeQueue cq : maps.values()) {
 28:                 boolean result = false;
 29:                 for (int i = 0; i < retryTimes && !result; i++) {
 30:                     result = cq.flush(flushConsumeQueueLeastPages);
 31:                 }
 32:             }
 33:         }
 34:         // flush Storage Checkpoint
 35:         if (0 == flushConsumeQueueLeastPages) {
 36:             if (logicsMsgTimestamp > 0) {
 37:                 DefaultMessageStore.this.getStoreCheckpoint().setLogicsMsgTimestamp(logicsMsgTimestamp);
 38:             }
 39:             DefaultMessageStore.this.getStoreCheckpoint().flush();
 40:         }
 41:     }
 42: 
 43:     public void run() {
 44:         DefaultMessageStore.log.info(this.getServiceName() + " service started");
 45: 
 46:         while (!this.isStopped()) {
 47:             try {
 48:                 int interval = DefaultMessageStore.this.getMessageStoreConfig().getFlushIntervalConsumeQueue();
 49:                 this.waitForRunning(interval);
 50:                 this.doFlush(1);
 51:             } catch (Exception e) {
 52:                 DefaultMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
 53:             }
 54:         }
 55: 
 56:         this.doFlush(RETRY_TIMES_OVER);
 57: 
 58:         DefaultMessageStore.log.info(this.getServiceName() + " service end");
 59:     }
 60: 
 61:     @Override
 62:     public String getServiceName() {
 63:         return FlushConsumeQueueService.class.getSimpleName();
 64:     }
 65: 
 66:     @Override
 67:     public long getJointime() {
 68:         return 1000 * 60;
 69:     }
 70: }

Description: Flush ConsumeQueue (Consumption Queue) Thread Service.
Lines 11 to 14: Force flush when retryTimes = RETRY_TIMES_OVER. For shutdown.
Lines 15 to 23: Execute flush once per flush Consume Queue Thorough Interval cycle. Because the size of the flush Consume Queue Least Pages is not always satisfied every time a loop arrives, a mandatory flush is required for a certain period of time. Of course, mandatory flush cannot be executed every time a loop, which results in poor performance.
Lines 24 to 33: flush Consume Queue.
- flush logic: MappedFile# Settlement.
Lines 34 to 40: flush Store Checkpoint. For a detailed analysis of StoreCheckpoint, see: Store Initialization and Closing.
Lines 43 to 59: Flush is executed every 1000ms. If wakeup(), a flush is performed immediately. At present, there is no wakeup() call.

4. Broker provides [pull-and-cancel] interface

PullMessageRequestHeader

  1: public class PullMessageRequestHeader implements CommandCustomHeader {
  2:     /**
  3:      * Consumer grouping
  4:      */
  5:     @CFNotNull
  6:     private String consumerGroup;
  7:     /**
  8:      * Topic
  9:      */
 10:     @CFNotNull
 11:     private String topic;
 12:     /**
 13:      * Queue number
 14:      */
 15:     @CFNotNull
 16:     private Integer queueId;
 17:     /**
 18:      * Queue start position
 19:      */
 20:     @CFNotNull
 21:     private Long queueOffset;
 22:     /**
 23:      * Number of messages
 24:      */
 25:     @CFNotNull
 26:     private Integer maxMsgNums;
 27:     /**
 28:      * System identification
 29:      */
 30:     @CFNotNull
 31:     private Integer sysFlag;
 32:     /**
 33:      * Submit Consumption Progress Location
 34:      */
 35:     @CFNotNull
 36:     private Long commitOffset;
 37:     /**
 38:      * Hang-up timeout
 39:      */
 40:     @CFNotNull
 41:     private Long suspendTimeoutMillis;
 42:     /**
 43:      * Subscription expression
 44:      */
 45:     @CFNullable
 46:     private String subscription;
 47:     /**
 48:      * Subscription version number
 49:      */
 50:     @CFNotNull
 51:     private Long subVersion;
 52: }

Note: Pull-cancel Interest Request Header
topic + queueId + queueOffset + maxMsgNums
sysFlag: System ID.
- No. 0 FLAG_COMMIT_OFFSET: Mark the location of the request submission consumption schedule and cooperate with commitOffset.
- Bit 1 FLAG_SUSPEND: Mark whether the request is pending and cooperate with suspend Timeout Millis. When a message is not pulled, Broker suspends the request until there is a message. Maximum hang time: suspend Timeout Millis milliseconds.
- Bit 2 FLAG_SUBSCRIPTION: Whether to filter subscription expressions, and subscription configuration.
subVersion: Subscription number. When requested, if the version number is incorrect, the message cannot be retrieved, and the subscription information needs to be retrieved to use the latest subscription version number.

PullMessageProcessor#processRequest(...)

  1: private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
  2:     throws RemotingCommandException {
  3:     RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
  4:     final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
  5:     final PullMessageRequestHeader requestHeader =
  6:         (PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
  7: 
  8:     response.setOpaque(request.getOpaque());
  9: 
 10:     if (LOG.isDebugEnabled()) {
 11:         LOG.debug("receive PullMessage request command, {}", request);
 12:     }
 13: 
 14:     // Check whether broker is readable
 15:     if (!PermName.isReadable(this.brokerController.getBrokerConfig().getBrokerPermission())) {
 16:         response.setCode(ResponseCode.NO_PERMISSION);
 17:         response.setRemark(String.format("the broker[%s] pulling message is forbidden", this.brokerController.getBrokerConfig().getBrokerIP1()));
 18:         return response;
 19:     }
 20: 
 21:     // Check whether consumer packet configuration exists
 22:     SubscriptionGroupConfig subscriptionGroupConfig = this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getConsumerGroup());
 23:     if (null == subscriptionGroupConfig) {
 24:         response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST);
 25:         response.setRemark(String.format("subscription group [%s] does not exist, %s", requestHeader.getConsumerGroup(), FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST)));
 26:         return response;
 27:     }
 28:     // Check whether consumer group configuration is consumable
 29:     if (!subscriptionGroupConfig.isConsumeEnable()) {
 30:         response.setCode(ResponseCode.NO_PERMISSION);
 31:         response.setRemark("subscription group no permission, " + requestHeader.getConsumerGroup());
 32:         return response;
 33:     }
 34: 
 35:     final boolean hasSuspendFlag = PullSysFlag.hasSuspendFlag(requestHeader.getSysFlag()); // Whether to suspend the request when there is no message
 36:     final boolean hasCommitOffsetFlag = PullSysFlag.hasCommitOffsetFlag(requestHeader.getSysFlag()); // Whether to submit consumption schedule
 37:     final boolean hasSubscriptionFlag = PullSysFlag.hasSubscriptionFlag(requestHeader.getSysFlag()); // Whether to filter subscription expressions
 38:     final long suspendTimeoutMillisLong = hasSuspendFlag ? requestHeader.getSuspendTimeoutMillis() : 0; // Suspended request timeout
 39: 
 40:     // Check that topic configuration exists
 41:     TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
 42:     if (null == topicConfig) {
 43:         LOG.error("The topic {} not exist, consumer: {} ", requestHeader.getTopic(), RemotingHelper.parseChannelRemoteAddr(channel));
 44:         response.setCode(ResponseCode.TOPIC_NOT_EXIST);
 45:         response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(), FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
 46:         return response;
 47:     }
 48:     // Check that topic configuration permissions are readable
 49:     if (!PermName.isReadable(topicConfig.getPerm())) {
 50:         response.setCode(ResponseCode.NO_PERMISSION);
 51:         response.setRemark("the topic[" + requestHeader.getTopic() + "] pulling message is forbidden");
 52:         return response;
 53:     }
 54:     // Check read queue is within the top configuration queue
 55:     if (requestHeader.getQueueId() < 0 || requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) {
 56:         String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] consumer:[%s]",
 57:                 requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(), channel.remoteAddress());
 58:         LOG.warn(errorInfo);
 59:         response.setCode(ResponseCode.SYSTEM_ERROR);
 60:         response.setRemark(errorInfo);
 61:         return response;
 62:     }
 63: 
 64:     // Check subscription relationship
 65:     SubscriptionData subscriptionData;
 66:     if (hasSubscriptionFlag) {
 67:         try {
 68:             subscriptionData = FilterAPI.buildSubscriptionData(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
 69:                 requestHeader.getSubscription());
 70:         } catch (Exception e) {
 71:             LOG.warn("Parse the consumer's subscription[{}] failed, group: {}", requestHeader.getSubscription(), //
 72:                     requestHeader.getConsumerGroup());
 73:             response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
 74:             response.setRemark("parse the consumer's subscription failed");
 75:             return response;
 76:         }
 77:     } else {
 78:         // Check the existence of consumer grouping information
 79:         ConsumerGroupInfo consumerGroupInfo = this.brokerController.getConsumerManager().getConsumerGroupInfo(requestHeader.getConsumerGroup());
 80:         if (null == consumerGroupInfo) {
 81:             LOG.warn("The consumer's group info not exist, group: {}", requestHeader.getConsumerGroup());
 82:             response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
 83:             response.setRemark("the consumer's group info not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
 84:             return response;
 85:         }
 86:         // Check whether the message model of consumer grouping information matches
 87:         if (!subscriptionGroupConfig.isConsumeBroadcastEnable() //
 88:             && consumerGroupInfo.getMessageModel() == MessageModel.BROADCASTING) {
 89:             response.setCode(ResponseCode.NO_PERMISSION);
 90:             response.setRemark("the consumer group[" + requestHeader.getConsumerGroup() + "] can not consume by broadcast way");
 91:             return response;
 92:         }
 93: 
 94:         // Check whether subscription information exists
 95:         subscriptionData = consumerGroupInfo.findSubscriptionData(requestHeader.getTopic());
 96:         if (null == subscriptionData) {
 97:             LOG.warn("The consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), requestHeader.getTopic());
 98:             response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
 99:             response.setRemark("the consumer's subscription not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
100:             return response;
101:         }
102:         // Verify that the subscription information version is legitimate
103:         if (subscriptionData.getSubVersion() < requestHeader.getSubVersion()) {
104:             LOG.warn("The broker's subscription is not latest, group: {} {}", requestHeader.getConsumerGroup(),
105:                     subscriptionData.getSubString());
106:             response.setCode(ResponseCode.SUBSCRIPTION_NOT_LATEST);
107:             response.setRemark("the consumer's subscription not latest");
108:             return response;
109:         }
110:     }
111: 
112:     // Get the message
113:     final GetMessageResult getMessageResult = this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
114:             requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), subscriptionData);
115:     if (getMessageResult != null) {
116:         response.setRemark(getMessageResult.getStatus().name());
117:         responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset());
118:         responseHeader.setMinOffset(getMessageResult.getMinOffset());
119:         responseHeader.setMaxOffset(getMessageResult.getMaxOffset());
120: 
121:         // TODO to Read
122:         // Computing recommended reading brokerId
123:         if (getMessageResult.isSuggestPullingFromSlave()) {
124:             responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
125:         } else {
126:             responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
127:         }
128: 
129:         switch (this.brokerController.getMessageStoreConfig().getBrokerRole()) {
130:             case ASYNC_MASTER:
131:             case SYNC_MASTER:
132:                 break;
133:             case SLAVE:
134:                 if (!this.brokerController.getBrokerConfig().isSlaveReadEnable()) { // The slave node is not allowed to read, telling the consumer to read the master node.
135:                     response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
136:                     responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
137:                 }
138:                 break;
139:         }
140: 
141:         if (this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
142:             // consume too slow ,redirect to another machine
143:             if (getMessageResult.isSuggestPullingFromSlave()) {
144:                 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
145:             }
146:             // consume ok
147:             else {
148:                 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
149:             }
150:         } else {
151:             responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
152:         }
153: 
154:         switch (getMessageResult.getStatus()) {
155:             case FOUND:
156:                 response.setCode(ResponseCode.SUCCESS);
157:                 break;
158:             case MESSAGE_WAS_REMOVING:
159:                 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
160:                 break;
161:             case NO_MATCHED_LOGIC_QUEUE:
162:             case NO_MESSAGE_IN_QUEUE:
163:                 if (0 != requestHeader.getQueueOffset()) {
164:                     response.setCode(ResponseCode.PULL_OFFSET_MOVED);
165: 
166:                     // XXX: warn and notify me
167:                     LOG.info("the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}", //
168:                         requestHeader.getQueueOffset(), //
169:                         getMessageResult.getNextBeginOffset(), //
170:                         requestHeader.getTopic(), //
171:                         requestHeader.getQueueId(), //
172:                         requestHeader.getConsumerGroup()//
173:                     );
174:                 } else {
175:                     response.setCode(ResponseCode.PULL_NOT_FOUND);
176:                 }
177:                 break;
178:             case NO_MATCHED_MESSAGE:
179:                 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
180:                 break;
181:             case OFFSET_FOUND_NULL:
182:                 response.setCode(ResponseCode.PULL_NOT_FOUND);
183:                 break;
184:             case OFFSET_OVERFLOW_BADLY:
185:                 response.setCode(ResponseCode.PULL_OFFSET_MOVED);
186:                 // XXX: warn and notify me
187:                 LOG.info("The request offset:{} over flow badly, broker max offset:{} , consumer: {}", requestHeader.getQueueOffset(), getMessageResult.getMaxOffset(), channel.remoteAddress());
188:                 break;
189:             case OFFSET_OVERFLOW_ONE:
190:                 response.setCode(ResponseCode.PULL_NOT_FOUND);
191:                 break;
192:             case OFFSET_TOO_SMALL:
193:                 response.setCode(ResponseCode.PULL_OFFSET_MOVED);
194:                 LOG.info("The request offset is too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}",
195:                     requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(),
196:                     getMessageResult.getMinOffset(), channel.remoteAddress());
197:                 break;
198:             default:
199:                 assert false;
200:                 break;
201:         }
202: 
203:         // hook: before
204:         if (this.hasConsumeMessageHook()) {
205:             ConsumeMessageContext context = new ConsumeMessageContext();
206:             context.setConsumerGroup(requestHeader.getConsumerGroup());
207:             context.setTopic(requestHeader.getTopic());
208:             context.setQueueId(requestHeader.getQueueId());
209: 
210:             String owner = request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER);
211: 
212:             switch (response.getCode()) {
213:                 case ResponseCode.SUCCESS:
214:                     int commercialBaseCount = brokerController.getBrokerConfig().getCommercialBaseCount();
215:                     int incValue = getMessageResult.getMsgCount4Commercial() * commercialBaseCount;
216: 
217:                     context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_SUCCESS);
218:                     context.setCommercialRcvTimes(incValue);
219:                     context.setCommercialRcvSize(getMessageResult.getBufferTotalSize());
220:                     context.setCommercialOwner(owner);
221: 
222:                     break;
223:                 case ResponseCode.PULL_NOT_FOUND:
224:                     if (!brokerAllowSuspend) {
225: 
226:                         context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
227:                         context.setCommercialRcvTimes(1);
228:                         context.setCommercialOwner(owner);
229: 
230:                     }
231:                     break;
232:                 case ResponseCode.PULL_RETRY_IMMEDIATELY:
233:                 case ResponseCode.PULL_OFFSET_MOVED:
234:                     context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
235:                     context.setCommercialRcvTimes(1);
236:                     context.setCommercialOwner(owner);
237:                     break;
238:                 default:
239:                     assert false;
240:                     break;
241:             }
242: 
243:             this.executeConsumeMessageHookBefore(context);
244:         }
245: 
246:         switch (response.getCode()) {
247:             case ResponseCode.SUCCESS:
248: 
249:                 this.brokerController.getBrokerStatsManager().incGroupGetNums(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
250:                     getMessageResult.getMessageCount());
251:                 this.brokerController.getBrokerStatsManager().incGroupGetSize(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
252:                     getMessageResult.getBufferTotalSize());
253:                 this.brokerController.getBrokerStatsManager().incBrokerGetNums(getMessageResult.getMessageCount());
254:                 // Read messages
255:                 if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) { // In memory
256:                     final long beginTimeMills = this.brokerController.getMessageStore().now();
257: 
258:                     final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
259: 
260:                     this.brokerController.getBrokerStatsManager().incGroupGetLatency(requestHeader.getConsumerGroup(),
261:                         requestHeader.getTopic(), requestHeader.getQueueId(),
262:                         (int) (this.brokerController.getMessageStore().now() - beginTimeMills));
263:                     response.setBody(r);
264:                 } else { // zero-copy
265:                     try {
266:                         FileRegion fileRegion = new ManyMessageTransfer(response.encodeHeader(getMessageResult.getBufferTotalSize()), getMessageResult);
267:                         channel.writeAndFlush(fileRegion).addListener(new ChannelFutureListener() {
268:                             @Override
269:                             public void operationComplete(ChannelFuture future) throws Exception {
270:                                 getMessageResult.release();
271:                                 if (!future.isSuccess()) {
272:                                     LOG.error("Fail to transfer messages from page cache to {}", channel.remoteAddress(), future.cause());
273:                                 }
274:                             }
275:                         });
276:                     } catch (Throwable e) {
277:                         LOG.error("Error occurred when transferring messages from page cache", e);
278:                         getMessageResult.release();
279:                     }
280: 
281:                     response = null;
282:                 }
283:                 break;
284:             case ResponseCode.PULL_NOT_FOUND:
285:                 // Message not queried & broker allows pending requests & requests allow pending
286:                 if (brokerAllowSuspend && hasSuspendFlag) {
287:                     long pollingTimeMills = suspendTimeoutMillisLong;
288:                     if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
289:                         pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
290:                     }
291: 
292:                     String topic = requestHeader.getTopic();
293:                     long offset = requestHeader.getQueueOffset();
294:                     int queueId = requestHeader.getQueueId();
295:                     PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
296:                         this.brokerController.getMessageStore().now(), offset, subscriptionData);
297:                     this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
298:                     response = null;
299:                     break;
300:                 }
301: 
302:             case ResponseCode.PULL_RETRY_IMMEDIATELY:
303:                 break;
304:             case ResponseCode.PULL_OFFSET_MOVED:
305:                 if (this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE
306:                     || this.brokerController.getMessageStoreConfig().isOffsetCheckInSlave()) { // TODO to be added to the blog
307:                     MessageQueue mq = new MessageQueue();
308:                     mq.setTopic(requestHeader.getTopic());
309:                     mq.setQueueId(requestHeader.getQueueId());
310:                     mq.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName());
311: 
312:                     OffsetMovedEvent event = new OffsetMovedEvent();
313:                     event.setConsumerGroup(requestHeader.getConsumerGroup());
314:                     event.setMessageQueue(mq);
315:                     event.setOffsetRequest(requestHeader.getQueueOffset());
316:                     event.setOffsetNew(getMessageResult.getNextBeginOffset());
317:                     this.generateOffsetMovedEvent(event);
318:                     LOG.warn(
319:                         "PULL_OFFSET_MOVED:correction offset. topic={}, groupId={}, requestOffset={}, newOffset={}, suggestBrokerId={}",
320:                         requestHeader.getTopic(), requestHeader.getConsumerGroup(), event.getOffsetRequest(), event.getOffsetNew(),
321:                         responseHeader.getSuggestWhichBrokerId());
322:                 } else {
323:                     responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
324:                     response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
325:                     LOG.warn("PULL_OFFSET_MOVED:none correction. topic={}, groupId={}, requestOffset={}, suggestBrokerId={}",
326:                         requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getQueueOffset(),
327:                         responseHeader.getSuggestWhichBrokerId());
328:                 }
329: 
330:                 break;
331:             default:
332:                 assert false;
333:         }
334:     } else {
335:         response.setCode(ResponseCode.SYSTEM_ERROR);
336:         response.setRemark("store getMessage return null");
337:     }
338: 
339:     // The request requests a persistence schedule & broker is not the owner, making the persistence schedule.
340:     boolean storeOffsetEnable = brokerAllowSuspend;
341:     storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag;
342:     storeOffsetEnable = storeOffsetEnable && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;
343:     if (storeOffsetEnable) {
344:         this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel),
345:             requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
346:     }
347:     return response;
348: }

Description: Processing pull-cancel request and returning response.
Lines 14 to 19: Check whether Broker is readable.
Lines 21 to 33: Check whether Subscription Group Config exists & can be consumed.
Lines 35 to 38: Handle the corresponding flag bit for PullMessageRequestHeader.sysFlag.
Lines 40 to 62: Check whether TopicConfig exists & readable & queue number is correct.
Lines 64 to 110: Check that Subscription Data (subscription information) is correct.
Line 113: Call MessageStore getMessage (...) to get GetMessageResult. For detailed analysis, see: MessageStore#getMessage(...).
Lines 122 to 152: Calculate the recommended pull-out rate brokerId.
From 154 to 201:
Lines 204 to 244: Hook logic, executeConsumeMessageHookBefore(...).
Lines 247 to 283: Successful pull-out, i.e. pull-out message.
- Lines 255 to 263: Mode 1: Call readGetMessageResult(...) to get the message content into the heap memory and set it to the response body.
- Lines 265 to 281: Mode 2: zero-copy based implementation, direct response, no need for in-heap memory, better performance. TODO: Here we have a study of zero-copy, and add a few more.
Lines 284 to 300: Pull out the message and execute the pending request when the condition is met (Broker allows pending & request pending). For detailed analysis, see: PullRequestHoldService.
Lines 304 to 328: TODO: This is a supplement to the tools module study.
339-346: Persistent consumption progress, when satisfied (Broker non-owner & request request for persistent progress). For detailed analysis, see: Update Consumption Progress.

MessageStore#getMessage(...)

  1: /**
  2:  * Get message results
  3:  *
  4:  * @param group Consumption grouping
  5:  * @param topic theme
  6:  * @param queueId Queue number
  7:  * @param offset Queue position
  8:  * @param maxMsgNums Number of messages
  9:  * @param subscriptionData Subscription information
 10:  * @return Message results
 11:  */
 12: public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset, final int maxMsgNums,
 13:     final SubscriptionData subscriptionData) {
 14:     // Whether to close or not
 15:     if (this.shutdown) {
 16:         log.warn("message store has shutdown, so getMessage is forbidden");
 17:         return null;
 18:     }
 19:     // Readability
 20:     if (!this.runningFlags.isReadable()) {
 21:         log.warn("message store is not readable, so getMessage is forbidden " + this.runningFlags.getFlagBits());
 22:         return null;
 23:     }
 24: 
 25:     long beginTime = this.getSystemClock().now();
 26: 
 27:     GetMessageStatus status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
 28:     long nextBeginOffset = offset;
 29:     long minOffset = 0;
 30:     long maxOffset = 0;
 31: 
 32:     GetMessageResult getResult = new GetMessageResult();
 33: 
 34:     final long maxOffsetPy = this.commitLog.getMaxOffset();
 35: 
 36:     // Obtain consumption queue
 37:     ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
 38:     if (consumeQueue != null) {
 39:         minOffset = consumeQueue.getMinOffsetInQueue(); // Minimum queue number of consumption queue
 40:         maxOffset = consumeQueue.getMaxOffsetInQueue(); // Maximum queue number for consumption queue
 41: 
 42:         // Judging queue position (offset)
 43:         if (maxOffset == 0) { // Consumption queue no news
 44:             status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
 45:             nextBeginOffset = nextOffsetCorrection(offset, 0);
 46:         } else if (offset < minOffset) { // Query offset is too small
 47:             status = GetMessageStatus.OFFSET_TOO_SMALL;
 48:             nextBeginOffset = nextOffsetCorrection(offset, minOffset);
 49:         } else if (offset == maxOffset) { // Query offset for more than one location in the consumer queue
 50:             status = GetMessageStatus.OFFSET_OVERFLOW_ONE;
 51:             nextBeginOffset = nextOffsetCorrection(offset, offset);
 52:         } else if (offset > maxOffset) { // Query offset over consumption queue too much (greater than one location)
 53:             status = GetMessageStatus.OFFSET_OVERFLOW_BADLY;
 54:             if (0 == minOffset) { // TODO blog, where is it? Why did 0 = minOffset make a special judgement?
 55:                 nextBeginOffset = nextOffsetCorrection(offset, minOffset);
 56:             } else {
 57:                 nextBeginOffset = nextOffsetCorrection(offset, maxOffset);
 58:             }
 59:         } else {
 60:             // Get the Mapped Buffer result (MappedFile)
 61:             SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset);
 62:             if (bufferConsumeQueue != null) {
 63:                 try {
 64:                     status = GetMessageStatus.NO_MATCHED_MESSAGE;
 65: 
 66:                     long nextPhyFileStartOffset = Long.MIN_VALUE; // The start offset corresponding to the next MappedFile in commitLog.
 67:                     long maxPhyOffsetPulling = 0; // Maximum offset pulled from the physical location of the message
 68: 
 69:                     int i = 0;
 70:                     final int maxFilterMessageCount = 16000;
 71:                     final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
 72:                     // Recycling for message location information
 73:                     for (; i < bufferConsumeQueue.getSize() && i < maxFilterMessageCount; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
 74:                         long offsetPy = bufferConsumeQueue.getByteBuffer().getLong(); // Message physical location offset
 75:                         int sizePy = bufferConsumeQueue.getByteBuffer().getInt(); // Message length
 76:                         long tagsCode = bufferConsumeQueue.getByteBuffer().getLong(); // Message tagsCode
 77:                         // Set the maximum offset pulled from the physical location of the message
 78:                         maxPhyOffsetPulling = offsetPy;
 79:                         // When offsetPy is less than nextPhyFileStartOffset, it means that the corresponding message has been removed, so continue directly until the readable Message.
 80:                         if (nextPhyFileStartOffset != Long.MIN_VALUE) {
 81:                             if (offsetPy < nextPhyFileStartOffset)
 82:                                 continue;
 83:                         }
 84:                         // Verify that commitLog requires a hard disk and cannot be stored in memory
 85:                         boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
 86:                         // Have you got enough information?
 87:                         if (this.isTheBatchFull(sizePy, maxMsgNums, getResult.getBufferTotalSize(), getResult.getMessageCount(),
 88:                             isInDisk)) {
 89:                             break;
 90:                         }
 91:                         // Determine whether the message meets the criteria
 92:                         if (this.messageFilter.isMessageMatched(subscriptionData, tagsCode)) {
 93:                             // Getting the corresponding message ByteBuffer from commitLog
 94:                             SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
 95:                             if (selectResult != null) {
 96:                                 this.storeStatsService.getGetMessageTransferedMsgCount().incrementAndGet();
 97:                                 getResult.addMessage(selectResult);
 98:                                 status = GetMessageStatus.FOUND;
 99:                                 nextPhyFileStartOffset = Long.MIN_VALUE;
100:                             } else {
101:                                 // The message cannot be read from commitLog, indicating that the corresponding file of the message (MappedFile) has been deleted and calculating the starting position of the next MappedFile
102:                                 if (getResult.getBufferTotalSize() == 0) {
103:                                     status = GetMessageStatus.MESSAGE_WAS_REMOVING;
104:                                 }
105:                                 nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy);
106:                             }
107:                         } else {
108:                             if (getResult.getBufferTotalSize() == 0) {
109:                                 status = GetMessageStatus.NO_MATCHED_MESSAGE;
110:                             }
111: 
112:                             if (log.isDebugEnabled()) {
113:                                 log.debug("message type not matched, client: " + subscriptionData + " server: " + tagsCode);
114:                             }
115:                         }
116:                     }
117:                     // Statistical Remaining Number of Kola Cancellation Interest Bytes
118:                     if (diskFallRecorded) {
119:                         long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
120:                         brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
121:                     }
122:                     // Calculate the message queue number of the next pull-out message
123:                     nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
124:                     // Judging whether it is recommended to read slave nodes based on the number of bytes and memory of the remaining Kola cancellations
125:                     long diff = maxOffsetPy - maxPhyOffsetPulling;
126:                     long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE
127:                             * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
128:                     getResult.setSuggestPullingFromSlave(diff > memory);
129:                 } finally {
130:                     bufferConsumeQueue.release();
131:                 }
132:             } else {
133:                 status = GetMessageStatus.OFFSET_FOUND_NULL;
134:                 nextBeginOffset = nextOffsetCorrection(offset, consumeQueue.rollNextFile(offset));
135:                 log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
136:                     + maxOffset + ", but access logic queue failed.");
137:             }
138:         }
139:     } else {
140:         status = GetMessageStatus.NO_MATCHED_LOGIC_QUEUE;
141:         nextBeginOffset = nextOffsetCorrection(offset, 0);
142:     }
143:     // Statistics
144:     if (GetMessageStatus.FOUND == status) {
145:         this.storeStatsService.getGetMessageTimesTotalFound().incrementAndGet();
146:     } else {
147:         this.storeStatsService.getGetMessageTimesTotalMiss().incrementAndGet();
148:     }
149:     long eclipseTime = this.getSystemClock().now() - beginTime;
150:     this.storeStatsService.setGetMessageEntireTimeMax(eclipseTime);
151:     // Set the return result
152:     getResult.setStatus(status);
153:     getResult.setNextBeginOffset(nextBeginOffset);
154:     getResult.setMaxOffset(maxOffset);
155:     getResult.setMinOffset(minOffset);
156:     return getResult;
157: }
158: 
159: /**
160:  * Get consumption queues based on topic + queue number
161:  *
162:  * @param topic theme
163:  * @param queueId Queue number
164:  * @return Consumption queue
165:  */
166: public ConsumeQueue findConsumeQueue(String topic, int queueId) {
167:     // Get all consumption queues corresponding to topic
168:     ConcurrentHashMap<Integer, ConsumeQueue> map = consumeQueueTable.get(topic);
169:     if (null == map) {
170:         ConcurrentHashMap<Integer, ConsumeQueue> newMap = new ConcurrentHashMap<>(128);
171:         ConcurrentHashMap<Integer, ConsumeQueue> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
172:         if (oldMap != null) {
173:             map = oldMap;
174:         } else {
175:             map = newMap;
176:         }
177:     }
178:     // Get the queue corresponding to queueId
179:     ConsumeQueue logic = map.get(queueId);
180:     if (null == logic) {
181:         ConsumeQueue newLogic = new ConsumeQueue(//
182:             topic, //
183:             queueId, //
184:             StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()), //
185:             this.getMessageStoreConfig().getMapedFileSizeConsumeQueue(), //
186:             this);
187:         ConsumeQueue oldLogic = map.putIfAbsent(queueId, newLogic);
188:         if (oldLogic != null) {
189:             logic = oldLogic;
190:         } else {
191:             logic = newLogic;
192:         }
193:     }
194: 
195:     return logic;
196: }
197: 
198: /**
199:  * Next acquisition queue offset correction
200:  * Modified condition: Check offset switch is turned on by master node or slave node
201:  *
202:  * @param oldOffset Old queue offset
203:  * @param newOffset New queue offset
204:  * @return Modified queue offset
205:  */
206: private long nextOffsetCorrection(long oldOffset, long newOffset) {
207:     long nextOffset = oldOffset;
208:     if (this.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE || this.getMessageStoreConfig().isOffsetCheckInSlave()) {
209:         nextOffset = newOffset;
210:     }
211:     return nextOffset;
212: }
213: 
214: /**
215:  * Verify that commitLog requires a hard disk and cannot be stored in memory
216:  *
217:  * @param offsetPy commitLog Specify offset
218:  * @param maxOffsetPy commitLog Maximum offset
219:  * @return Do you need a hard disk?
220:  */
221: private boolean checkInDiskByCommitOffset(long offsetPy, long maxOffsetPy) {
222:     long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
223:     return (maxOffsetPy - offsetPy) > memory;
224: }
225: 
226: /**
227:  * Determine whether the retrieved message is full
228:  *
229:  * @param sizePy Number of bytes
230:  * @param maxMsgNums Maximum number of messages
231:  * @param bufferTotal The number of bytes has been calculated
232:  * @param messageTotal Number of messages has been calculated
233:  * @param isInDisk Is it on hard disk?
234:  * @return Is it full?
235:  */
236: private boolean isTheBatchFull(int sizePy, int maxMsgNums, int bufferTotal, int messageTotal, boolean isInDisk) {
237:     if (0 == bufferTotal || 0 == messageTotal) {
238:         return false;
239:     }
240:     // The number of messages has met the number of requests (maxMsgNums)
241:     if ((messageTotal + 1) >= maxMsgNums) {
242:         return true;
243:     }
244:     // Is the maximum number of transmitted bytes and messages full according to the message storage configuration?
245:     if (isInDisk) {
246:         if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInDisk()) {
247:             return true;
248:         }
249: 
250:         if ((messageTotal + 1) > this.messageStoreConfig.getMaxTransferCountOnMessageInDisk()) {
251:             return true;
252:         }
253:     } else {
254:         if ((bufferTotal + sizePy) > this.messageStoreConfig.getMaxTransferBytesOnMessageInMemory()) {
255:             return true;
256:         }
257: 
258:         if ((messageTotal + 1) > this.messageStoreConfig.getMaxTransferCountOnMessageInMemory()) {
259:             return true;
260:         }
261:     }
262: 
263:     return false;
264: }

Description: Get maxMsgNums messages according to message grouping + Topic + queue Id + offset + subscription data.
Lines 14 to 18: Determine whether the Store is closed, and if it is closed, no message can be obtained.
Lines 19 to 23: Determine whether the current running state is readable or not, and if not, no message can be obtained.
Line 37: Consume Queue based on Topic + queue Id.
- # findConsumeQueue(...): Lines 159 to 196.
Lines 43 to 58: Various queue locations (offsets) cannot read messages, and the next Client queue pull location is calculated for the corresponding situation.
- Lines 43 to 45: No messages in the message queue.
- Lines 46 to 48: The message queue location (offset) of the query is too small.
- Lines 49 to 51: The query's message queue location (offset) is exactly equal to the maximum queue location of the message queue. This is a normal phenomenon, which is equivalent to inquiring about the latest news.
- Lines 52 to 58: Excessive number of message queue locations (offset s) are queried.
- # Next Offset Correction (...): Lines 198 to 212.
Line 61: Get the corresponding MappedFile based on the offset.
Lines 72 to 128: Recycle message location information.
- Lines 74 to 76: Read the location information for each message.
- Lines 79 to 83: When offsetPy is less than nextPhyFileStartOffset, it means that The corresponding message has been removed, so continue directly until the readable Message.
- Lines 84 to 90: Determine whether enough information has been obtained.
  - # checkInDiskByCommitOffset(...): Lines 214 to 224.
  - # isTheBatchFull(...): Lines 226 to 264.
Line 92: Determine whether the message meets the criteria. For detailed analysis, see: DefaultMessageFilter#isMessageMatched(...).
Line 94: MappedByteBuffer to get the corresponding message from CommitLog.
Lines 95 to 99: Get the message MappedByteBuffer successfully.
Lines 100 to 106: Getting the message MappedByteBuffer failed. The message cannot be read from CommitLog, indicating that the corresponding file (MappedFile) of the message has been deleted, at which time the starting position of the next MappedFile is calculated. This logic needs to be understood together (lines 79 to 83).
Lines 117 to 120: Count the number of bytes left in Kola Cancellation Interest.
Line 123: Calculate the message queue number for the next pull-out message.
Lines 124 to 128: Determine whether it is recommended to read slave nodes based on the number of bytes and memory left in the cancel message.
Line 130: Release bufferConsumeQueue's pointing to MappedFile. Here MappedFile is a file in ConsumeQueue, not a file under CommitLog.
Lines 133 to 136: Get the consumer queue location (offset) and get the corresponding MappedFile empty. Calculate the corresponding location of the next MappedFile starting from offset for ConsumeQueue.
Lines 143 to 150: Record statistics: time consumed, number of pull-in/pull-out messages.
Lines 151 to 156: Set the return result and return it.

DefaultMessageFilter#isMessageMatched(...)

  1: public class DefaultMessageFilter implements MessageFilter {
  2: 
  3:     @Override
  4:     public boolean isMessageMatched(SubscriptionData subscriptionData, Long tagsCode) {
  5:         // Message tagsCode empty
  6:         if (tagsCode == null) {
  7:             return true;
  8:         }
  9:         // Subscription data blank
 10:         if (null == subscriptionData) {
 11:             return true;
 12:         }
 13:         // classFilter
 14:         if (subscriptionData.isClassFilterMode())
 15:             return true;
 16:         // Subscription expression matching
 17:         if (subscriptionData.getSubString().equals(SubscriptionData.SUB_ALL)) {
 18:             return true;
 19:         }
 20:         // Does the subscription data code array contain message tagsCode
 21:         return subscriptionData.getCodeSet().contains(tagsCode.intValue());
 22:     }
 23: 
 24: }

Description: Message filter is implemented by default.

PullRequestHoldService

  1: public class PullRequestHoldService extends ServiceThread {
  2: 
  3:     private static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
  4: 
  5:     private static final String TOPIC_QUEUEID_SEPARATOR = "@";
  6: 
  7:     private final BrokerController brokerController;
  8: 
  9:     private final SystemClock systemClock = new SystemClock();
 10:     /**
 11:      * Message filter
 12:      */
 13:     private final MessageFilter messageFilter = new DefaultMessageFilter();
 14:     /**
 15:      * Pull-cancel interest request set
 16:      */
 17:     private ConcurrentHashMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 18:             new ConcurrentHashMap<>(1024);
 19: 
 20:     public PullRequestHoldService(final BrokerController brokerController) {
 21:         this.brokerController = brokerController;
 22:     }
 23: 
 24:     /**
 25:      * Add pull-cancel pending request
 26:      *
 27:      * @param topic theme
 28:      * @param queueId Queue number
 29:      * @param pullRequest Cancellation of Interest Request
 30:      */
 31:     public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 32:         String key = this.buildKey(topic, queueId);
 33:         ManyPullRequest mpr = this.pullRequestTable.get(key);
 34:         if (null == mpr) {
 35:             mpr = new ManyPullRequest();
 36:             ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 37:             if (prev != null) {
 38:                 mpr = prev;
 39:             }
 40:         }
 41: 
 42:         mpr.addPullRequest(pullRequest);
 43:     }
 44: 
 45:     /**
 46:      * Create unique identifiers based on topic + queue number
 47:      *
 48:      * @param topic theme
 49:      * @param queueId Queue number
 50:      * @return key
 51:      */
 52:     private String buildKey(final String topic, final int queueId) {
 53:         StringBuilder sb = new StringBuilder();
 54:         sb.append(topic);
 55:         sb.append(TOPIC_QUEUEID_SEPARATOR);
 56:         sb.append(queueId);
 57:         return sb.toString();
 58:     }
 59: 
 60:     @Override
 61:     public void run() {
 62:         log.info("{} service started", this.getServiceName());
 63:         while (!this.isStopped()) {
 64:             try {
 65:                 // Set up different waiting time according to long or short rotation training
 66:                 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 67:                     this.waitForRunning(5 * 1000);
 68:                 } else {
 69:                     this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 70:                 }
 71:                 // Check if pending requests need to be notified
 72:                 long beginLockTimestamp = this.systemClock.now();
 73:                 this.checkHoldRequest();
 74:                 long costTime = this.systemClock.now() - beginLockTimestamp;
 75:                 if (costTime > 5 * 1000) {
 76:                     log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 77:                 }
 78:             } catch (Throwable e) {
 79:                 log.warn(this.getServiceName() + " service has exception. ", e);
 80:             }
 81:         }
 82: 
 83:         log.info("{} service end", this.getServiceName());
 84:     }
 85: 
 86:     @Override
 87:     public String getServiceName() {
 88:         return PullRequestHoldService.class.getSimpleName();
 89:     }
 90: 
 91:     /**
 92:      * Traverse pending requests to check if there are any requests that need to be notified.
 93:      */
 94:     private void checkHoldRequest() {
 95:         for (String key : this.pullRequestTable.keySet()) {
 96:             String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 97:             if (2 == kArray.length) {
 98:                 String topic = kArray[0];
 99:                 int queueId = Integer.parseInt(kArray[1]);
100:                 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQuque(topic, queueId);
101:                 try {
102:                     this.notifyMessageArriving(topic, queueId, offset);
103:                 } catch (Throwable e) {
104:                     log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
105:                 }
106:             }
107:         }
108:     }
109: 
110:     /**
111:      * Check for requests for notification
112:      *
113:      * @param topic theme
114:      * @param queueId Queue number
115:      * @param maxOffset Maximum offset of consumption queue
116:      */
117:     public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
118:         notifyMessageArriving(topic, queueId, maxOffset, null);
119:     }
120: 
121:     /**
122:      * Check for requests for notification
123:      *
124:      * @param topic theme
125:      * @param queueId Queue number
126:      * @param maxOffset Maximum offset of consumption queue
127:      * @param tagsCode Filtering tagsCode
128:      */
129:     public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode) {
130:         String key = this.buildKey(topic, queueId);
131:         ManyPullRequest mpr = this.pullRequestTable.get(key);
132:         if (mpr != null) {
133:             //
134:             List<PullRequest> requestList = mpr.cloneListAndClear();
135:             if (requestList != null) {
136:                 List<PullRequest> replayList = new ArrayList<>(); // An array of requests that do not conform to wake-up
137: 
138:                 for (PullRequest request : requestList) {
139:                     // If the maxOffset is too small, read it again.
140:                     long newestOffset = maxOffset;
141:                     if (newestOffset <= request.getPullFromThisOffset()) {
142:                         newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQuque(topic, queueId);
143:                     }
144:                     // There is a new matching message, wake-up request, that is, pull the cancel message again.
145:                     if (newestOffset > request.getPullFromThisOffset()) {
146:                         if (this.messageFilter.isMessageMatched(request.getSubscriptionData(), tagsCode)) {
147:                             try {
148:                                 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
149:                                     request.getRequestCommand());
150:                             } catch (Throwable e) {
151:                                 log.error("execute request when wakeup failed.", e);
152:                             }
153:                             continue;
154:                         }
155:                     }
156:                     // When the suspension time is exceeded, the request is awakened, i.e. the cancellation rate is pulled again.
157:                     if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
158:                         try {
159:                             this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
160:                                 request.getRequestCommand());
161:                         } catch (Throwable e) {
162:                             log.error("execute request when wakeup failed.", e);
163:                         }
164:                         continue;
165:                     }
166:                     // Do not comply with the request to retrieve again, add back again
167:                     replayList.add(request);
168:                 }
169:                 // Add back
170:                 if (!replayList.isEmpty()) {
171:                     mpr.addPullRequest(replayList);
172:                 }
173:             }
174:         }
175:     }
176: }

PullRequestHoldService Description: Pull cancel message request to suspend maintenance thread service.
- When the pull-out request cannot get the message, the request is suspended and added to the service.
- Re-execute the fetch message logic when there is qualified information or when the suspension timeout occurs.
# SusndPullRequest (...) Description: Add pull-cancel pending request to collection (pullRequestTable).
# run(...) Note: Check periodically whether the pending request needs to be notified to re-pull the cancellation rate and notify.
- Lines 65 to 70: Set different waiting times according to long or short rotation training.
- Lines 71 to 77: Check if there is a need for notification of pending requests.
# checkHoldRequest(...) specifies: traverse pending requests to check for notifications.
# NotfyMessageArriving (...) Notes: Check whether the specified queue has a request for notification.
- Lines 139 to 143: If maxOffset is too small, retrieve the latest.
- Lines 144 to 155: New matching message, wake-up request, that is, pull the cancel rate again.
- Lines 156 to 165: Wake-up request, i.e. pull the cancellation rate again, after the hang-up time.
- Line 148 | 159: Wake up the request and pull the cancellation rate again. I was afraid that the long time of pull-out would affect the whole traversal of pending requests. Then I look at # executeRequestWhenWakeup(...), which is actually a step-by-step message pull thrown into the thread pool. There will be no performance problems. For detailed analysis, see: PullMessageProcessor#executeRequestWhenWakeup(...).
- Lines 166 to 172: Requests that do not conform to wake-up are re-added to the collection (pullRequestTable).

PullMessageProcessor#executeRequestWhenWakeup(...)

  1: public void executeRequestWhenWakeup(final Channel channel, final RemotingCommand request) throws RemotingCommandException {
  2:     Runnable run = new Runnable() {
  3:         @Override
  4:         public void run() {
  5:             try {
  6:                 // Call pull request. For this call, set no pending request.
  7:                 final RemotingCommand response = PullMessageProcessor.this.processRequest(channel, request, false);
  8: 
  9:                 if (response != null) {
 10:                     response.setOpaque(request.getOpaque());
 11:                     response.markResponseType();
 12:                     try {
 13:                         channel.writeAndFlush(response).addListener(new ChannelFutureListener() {
 14:                             @Override
 15:                             public void operationComplete(ChannelFuture future) throws Exception {
 16:                                 if (!future.isSuccess()) {
 17:                                     LOG.error("ProcessRequestWrapper response to {} failed", future.channel().remoteAddress(), future.cause());
 18:                                     LOG.error(request.toString());
 19:                                     LOG.error(response.toString());
 20:                                 }
 21:                             }
 22:                         });
 23:                     } catch (Throwable e) {
 24:                         LOG.error("ProcessRequestWrapper process request over, but response failed", e);
 25:                         LOG.error(request.toString());
 26:                         LOG.error(response.toString());
 27:                     }
 28:                 }
 29:             } catch (RemotingCommandException e1) {
 30:                 LOG.error("ExecuteRequestWhenWakeup run", e1);
 31:             }
 32:         }
 33:     };
 34:     // Submit pull request to thread pool
 35:     this.brokerController.getPullMessageExecutor().submit(new RequestTask(run, channel, request));
 36: }

Description: Execution request wake-up, that is, pull the cancellation rate again. This method calls the thread pool, so it does not block.
Line 7: Call the pull-out request. In this call, the request is not suspended even if the message cannot be requested. If not set, the request may be suspended indefinitely and looped indefinitely by Broker.
Line 35: Submit the pull-out request to the thread pool.

5. Broker provides [update consumption schedule] interface

Yunai-MacdeMacBook-Pro-2:config yunai$ pwd
/Users/yunai/store/config
Yunai-MacdeMacBook-Pro-2:config yunai$ ls -ls
total 40
8 -rw-r--r--  1 yunai  staff    21  4 28 16:58 consumerOffset.json
8 -rw-r--r--  1 yunai  staff    21  4 28 16:58 consumerOffset.json.bak
8 -rw-r--r--  1 yunai  staff    21  4 28 16:58 delayOffset.json
8 -rw-r--r--  1 yunai  staff    21  4 28 16:58 delayOffset.json.bak
8 -rw-r--r--  1 yunai  staff  1401  4 27 21:51 topics.json
Yunai-MacdeMacBook-Pro-2:config yunai$ cat consumerOffset.json
{
    "offsetTable":{
        "%RETRY%please_rename_unique_group_name_4@please_rename_unique_group_name_4":{0:0
        },
        "TopicRead3@please_rename_unique_group_name_4":{1:5
        }
    }
}

consumerOffset.json: Consumption progress storage file.
ConsumererOffset. json. bak: Backup of the consumption progress storage file.
Each time consumerOffset.json is written, the original content is backed up to consumerOffset.json.bak. See: MixAll#string2File(...).

BrokerController#initialize(...)

  1:this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
  2:    @Override
  3:    public void run() {
  4:        try {
  5:            BrokerController.this.consumerOffsetManager.persist();
  6:        } catch (Throwable e) {
  7:            log.error("schedule persist consumerOffset error.", e);
  8:        }
  9:    }
 10:}, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

Description: Perform persistence logic every 5s.

ConfigManager

  1: public abstract class ConfigManager {
  2: private static final Logger PLOG = LoggerFactory.getLogger(LoggerName.COMMON_LOGGER_NAME);
  3: 
  4: /**
  5:  * Coded content
  6:  * @return Encoded content
  7:  */
  8: public abstract String encode();
  9: 
 10: /**
 11:  * load file
 12:  *
 13:  * @return Successful Loading
 14:  */
 15: public boolean load() {
 16:     String fileName = null;
 17:     try {
 18:         fileName = this.configFilePath();
 19:         String jsonString = MixAll.file2String(fileName);
 20:         // If the content does not exist, load the backup file
 21:         if (null == jsonString || jsonString.length() == 0) {
 22:             return this.loadBak();
 23:         } else {
 24:             this.decode(jsonString);
 25:             PLOG.info("load {} OK", fileName);
 26:             return true;
 27:         }
 28:     } catch (Exception e) {
 29:         PLOG.error("load " + fileName + " Failed, and try to load backup file", e);
 30:         return this.loadBak();
 31:     }
 32: }
 33: 
 34: /**
 35:  * Configuration file address
 36:  *
 37:  * @return Configuration file address
 38:  */
 39: public abstract String configFilePath();
 40: 
 41: /**
 42:  * Loading backup files
 43:  *
 44:  * @return Success or not
 45:  */
 46: private boolean loadBak() {
 47:     String fileName = null;
 48:     try {
 49:         fileName = this.configFilePath();
 50:         String jsonString = MixAll.file2String(fileName + ".bak");
 51:         if (jsonString != null && jsonString.length() > 0) {
 52:             this.decode(jsonString);
 53:             PLOG.info("load " + fileName + " OK");
 54:             return true;
 55:         }
 56:     } catch (Exception e) {
 57:         PLOG.error("load " + fileName + " Failed", e);
 58:         return false;
 59:     }
 60: 
 61:     return true;
 62: }
 63: 
 64: /**
 65:  * Decoded content
 66:  *
 67:  * @param jsonString content
 68:  */
 69: public abstract void decode(final String jsonString);
 70: 
 71: /**
 72:  * Persistence
 73:  */
 74: public synchronized void persist() {
 75:     String jsonString = this.encode(true);
 76:     if (jsonString != null) {
 77:         String fileName = this.configFilePath();
 78:         try {
 79:             MixAll.string2File(jsonString, fileName);
 80:         } catch (IOException e) {
 81:             PLOG.error("persist file Exception, " + fileName, e);
 82:         }
 83:     }
 84: }
 85: 
 86: /**
 87:  * Coded storage content
 88:  *
 89:  * @param prettyFormat Formatting or not
 90:  * @return content
 91:  */
 92: public abstract String encode(final boolean prettyFormat);
 93: }

MixAll#string2File(...)

  1: /**
  2:  * Write content to a file
  3:  * Safe Writing
  4:  * 1. Write to. tmp file
  5:  * 2. Backup ready to write to. bak file
  6:  * 3. Delete the original file and change. tmp to a file
  7:  *
  8:  * @param str content
  9:  * @param fileName file name
 10:  * @throws IOException When IO is abnormal
 11:  */
 12: public static void string2File(final String str, final String fileName) throws IOException {
 13:     // Write to tmp file
 14:     String tmpFile = fileName + ".tmp";
 15:     string2FileNotSafe(str, tmpFile);
 16:     //
 17:     String bakFile = fileName + ".bak";
 18:     String prevContent = file2String(fileName);
 19:     if (prevContent != null) {
 20:         string2FileNotSafe(prevContent, bakFile);
 21:     }
 22: 
 23:     File file = new File(fileName);
 24:     file.delete();
 25: 
 26:     file = new File(tmpFile);
 27:     file.renameTo(new File(fileName));
 28: }
 29: 
 30: /**
 31:  * Write content to a file
 32:  * Unsafe Writing
 33:  *
 34:  * @param str content
 35:  * @param fileName Document content
 36:  * @throws IOException When IO is abnormal
 37:  */
 38: public static void string2FileNotSafe(final String str, final String fileName) throws IOException {
 39:     File file = new File(fileName);
 40:     // Create a superior directory
 41:     File fileParent = file.getParentFile();
 42:     if (fileParent != null) {
 43:         fileParent.mkdirs();
 44:     }
 45:     // Writing Content
 46:     FileWriter fileWriter = null;
 47:     try {
 48:         fileWriter = new FileWriter(file);
 49:         fileWriter.write(str);
 50:     } catch (IOException e) {
 51:         throw e;
 52:     } finally {
 53:         if (fileWriter != null) {
 54:             fileWriter.close();
 55:         }
 56:     }
 57: }

ConsumerOffsetManager

  1: public class ConsumerOffsetManager extends ConfigManager {
  2:     private static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
  3:     private static final String TOPIC_GROUP_SEPARATOR = "@";
  4: 
  5:     /**
  6:      * Collection of Consumption Schedules
  7:      */
  8:     private ConcurrentHashMap<String/* topic@group */, ConcurrentHashMap<Integer, Long>> offsetTable = new ConcurrentHashMap<>(512);
  9: 
 10:     private transient BrokerController brokerController;
 11: 
 12:     public ConsumerOffsetManager() {
 13:     }
 14: 
 15:     public ConsumerOffsetManager(BrokerController brokerController) {
 16:         this.brokerController = brokerController;
 17:     }
 18: 
 19:     /**
 20:      * Submitting Consumption Progress
 21:      *
 22:      * @param clientHost Submit client address
 23:      * @param group Consumption grouping
 24:      * @param topic theme
 25:      * @param queueId Queue number
 26:      * @param offset Progress (queue location)
 27:      */
 28:     public void commitOffset(final String clientHost, final String group, final String topic, final int queueId, final long offset) {
 29:         // topic@group
 30:         String key = topic + TOPIC_GROUP_SEPARATOR + group;
 31:         this.commitOffset(clientHost, key, queueId, offset);
 32:     }
 33: 
 34:     /**
 35:      * Submitting Consumption Progress
 36:      *
 37:      * @param clientHost Submit client address
 38:      * @param key Subject @Consumption Grouping
 39:      * @param queueId Queue number
 40:      * @param offset Progress (queue location)
 41:      */
 42:     private void commitOffset(final String clientHost, final String key, final int queueId, final long offset) {
 43:         ConcurrentHashMap<Integer, Long> map = this.offsetTable.get(key);
 44:         if (null == map) {
 45:             map = new ConcurrentHashMap<>(32);
 46:             map.put(queueId, offset);
 47:             this.offsetTable.put(key, map);
 48:         } else {
 49:             Long storeOffset = map.put(queueId, offset);
 50:             if (storeOffset != null && offset < storeOffset) {
 51:                 log.warn("[NOTIFYME]update consumer offset less than store. clientHost={}, key={}, queueId={}, requestOffset={}, storeOffset={}", clientHost, key, queueId, offset, storeOffset);
 52:             }
 53:         }
 54:     }
 55: 
 56:     public String encode() {
 57:         return this.encode(false);
 58:     }
 59: 
 60:     @Override
 61:     public String configFilePath() {
 62:         return BrokerPathConfigHelper.getConsumerOffsetPath(this.brokerController.getMessageStoreConfig().getStorePathRootDir());
 63:     }
 64: 
 65:     /**
 66:      * Decoded content
 67:      * Format: JSON
 68:      *
 69:      * @param jsonString content
 70:      */
 71:     @Override
 72:     public void decode(String jsonString) {
 73:         if (jsonString != null) {
 74:             ConsumerOffsetManager obj = RemotingSerializable.fromJson(jsonString, ConsumerOffsetManager.class);
 75:             if (obj != null) {
 76:                 this.offsetTable = obj.offsetTable;
 77:             }
 78:         }
 79:     }
 80: 
 81:     /**
 82:      * Coded content
 83:      * Format JSON
 84:      *
 85:      * @param prettyFormat Formatting or not
 86:      * @return Encoded content
 87:      */
 88:     public String encode(final boolean prettyFormat) {
 89:         return RemotingSerializable.toJson(this, prettyFormat);
 90:     }
 91: 
 92: }

Description: Consumption Progress Manager.

6. Broker provides [send back message] interface

Most logical sum Broker provides [accepts messages] interfaces Similarly, you can look at the relevant content first.

SendMessageProcessor#consumerSendMsgBack(...)

  1: private RemotingCommand consumerSendMsgBack(final ChannelHandlerContext ctx, final RemotingCommand request)
  2:     throws RemotingCommandException {
  3: 
  4:     // Initialization response
  5:     final RemotingCommand response = RemotingCommand.createResponseCommand(null);
  6:     final ConsumerSendMsgBackRequestHeader requestHeader =
  7:         (ConsumerSendMsgBackRequestHeader) request.decodeCommandCustomHeader(ConsumerSendMsgBackRequestHeader.class);
  8: 
  9:     // hook (unique)
 10:     if (this.hasConsumeMessageHook() && !UtilAll.isBlank(requestHeader.getOriginMsgId())) {
 11: 
 12:         ConsumeMessageContext context = new ConsumeMessageContext();
 13:         context.setConsumerGroup(requestHeader.getGroup());
 14:         context.setTopic(requestHeader.getOriginTopic());
 15:         context.setCommercialRcvStats(BrokerStatsManager.StatsType.SEND_BACK);
 16:         context.setCommercialRcvTimes(1);
 17:         context.setCommercialOwner(request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER));
 18: 
 19:         this.executeConsumeMessageHookAfter(context);
 20:     }
 21: 
 22:     // Judging whether the consumption group exists (unique)
 23:     SubscriptionGroupConfig subscriptionGroupConfig =
 24:         this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getGroup());
 25:     if (null == subscriptionGroupConfig) {
 26:         response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST);
 27:         response.setRemark("subscription group not exist, " + requestHeader.getGroup() + " "
 28:             + FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST));
 29:         return response;
 30:     }
 31: 
 32:     // Check whether broker has write permission
 33:     if (!PermName.isWriteable(this.brokerController.getBrokerConfig().getBrokerPermission())) {
 34:         response.setCode(ResponseCode.NO_PERMISSION);
 35:         response.setRemark("the broker[" + this.brokerController.getBrokerConfig().getBrokerIP1() + "] sending message is forbidden");
 36:         return response;
 37:     }
 38: 
 39:     // Check if the number of retry queues is greater than 0 (unique)
 40:     if (subscriptionGroupConfig.getRetryQueueNums() <= 0) {
 41:         response.setCode(ResponseCode.SUCCESS);
 42:         response.setRemark(null);
 43:         return response;
 44:     }
 45: 
 46:     // Calculate retry Topic
 47:     String newTopic = MixAll.getRetryTopic(requestHeader.getGroup());
 48: 
 49:     // Calculate the queue number (unique)
 50:     int queueIdInt = Math.abs(this.random.nextInt() % 99999999) % subscriptionGroupConfig.getRetryQueueNums();
 51: 
 52:     // Computing sysFlag (unique)
 53:     int topicSysFlag = 0;
 54:     if (requestHeader.isUnitMode()) {
 55:         topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
 56:     }
 57: 
 58:     // Get topicConfig. If not, create
 59:     TopicConfig topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(//
 60:         newTopic, //
 61:         subscriptionGroupConfig.getRetryQueueNums(), //
 62:         PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
 63:     if (null == topicConfig) { // No configuration
 64:         response.setCode(ResponseCode.SYSTEM_ERROR);
 65:         response.setRemark("topic[" + newTopic + "] not exist");
 66:         return response;
 67:     }
 68:     if (!PermName.isWriteable(topicConfig.getPerm())) { // Writing is not allowed
 69:         response.setCode(ResponseCode.NO_PERMISSION);
 70:         response.setRemark(String.format("the topic[%s] sending message is forbidden", newTopic));
 71:         return response;
 72:     }
 73: 
 74:     // Query messages. If it does not exist, an exception error is returned. (unique)
 75:     MessageExt msgExt = this.brokerController.getMessageStore().lookMessageByOffset(requestHeader.getOffset());
 76:     if (null == msgExt) {
 77:         response.setCode(ResponseCode.SYSTEM_ERROR);
 78:         response.setRemark("look message by offset failed, " + requestHeader.getOffset());
 79:         return response;
 80:     }
 81: 
 82:     // Set retryTopic to Extended Properties (unique)
 83:     final String retryTopic = msgExt.getProperty(MessageConst.PROPERTY_RETRY_TOPIC);
 84:     if (null == retryTopic) {
 85:         MessageAccessor.putProperty(msgExt, MessageConst.PROPERTY_RETRY_TOPIC, msgExt.getTopic());
 86:     }
 87: 
 88:     // Setting messages not to wait for storage to complete (unique) TODO Question: If set to wait for storage, broker set to synchronize settings, can not be submitted in batches?
 89:     msgExt.setWaitStoreMsgOK(false);
 90: 
 91:     // Processing delayLevel (unique).
 92:     int delayLevel = requestHeader.getDelayLevel();
 93:     int maxReconsumeTimes = subscriptionGroupConfig.getRetryMaxTimes();
 94:     if (request.getVersion() >= MQVersion.Version.V3_4_9.ordinal()) {
 95:         maxReconsumeTimes = requestHeader.getMaxReconsumeTimes();
 96:     }
 97:     if (msgExt.getReconsumeTimes() >= maxReconsumeTimes//
 98:         || delayLevel < 0) { // If the maximum number of consumptions is exceeded, topic is changed to "% DLQ%" +group name, that is, to join the Dead Letter Queue.
 99:         newTopic = MixAll.getDLQTopic(requestHeader.getGroup());
100:         queueIdInt = Math.abs(this.random.nextInt() % 99999999) % DLQ_NUMS_PER_GROUP;
101: 
102:         topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(newTopic, //
103:             DLQ_NUMS_PER_GROUP, //
104:             PermName.PERM_WRITE, 0
105:         );
106:         if (null == topicConfig) {
107:             response.setCode(ResponseCode.SYSTEM_ERROR);
108:             response.setRemark("topic[" + newTopic + "] not exist");
109:             return response;
110:         }
111:     } else {
112:         if (0 == delayLevel) {
113:             delayLevel = 3 + msgExt.getReconsumeTimes();
114:         }
115:         msgExt.setDelayTimeLevel(delayLevel);
116:     }
117: 
118:     // Create Message Ext Broker Inner
119:     MessageExtBrokerInner msgInner = new MessageExtBrokerInner();
120:     msgInner.setTopic(newTopic);
121:     msgInner.setBody(msgExt.getBody());
122:     msgInner.setFlag(msgExt.getFlag());
123:     MessageAccessor.setProperties(msgInner, msgExt.getProperties());
124:     msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgExt.getProperties()));
125:     msgInner.setTagsCode(MessageExtBrokerInner.tagsString2tagsCode(null, msgExt.getTags()));
126:     msgInner.setQueueId(queueIdInt);
127:     msgInner.setSysFlag(msgExt.getSysFlag());
128:     msgInner.setBornTimestamp(msgExt.getBornTimestamp());
129:     msgInner.setBornHost(msgExt.getBornHost());
130:     msgInner.setStoreHost(this.getStoreHost());
131:     msgInner.setReconsumeTimes(msgExt.getReconsumeTimes() + 1);
132: 
133:     // Set the original message number to the extended field (unique)
134:     String originMsgId = MessageAccessor.getOriginMessageId(msgExt);
135:     MessageAccessor.setOriginMessageId(msgInner, UtilAll.isBlank(originMsgId) ? msgExt.getMsgId() : originMsgId);
136: 
137:     // Adding messages
138:     PutMessageResult putMessageResult = this.brokerController.getMessageStore().putMessage(msgInner);
139:     if (putMessageResult != null) {
140:         switch (putMessageResult.getPutMessageStatus()) {
141:             case PUT_OK:
142:                 String backTopic = msgExt.getTopic();
143:                 String correctTopic = msgExt.getProperty(MessageConst.PROPERTY_RETRY_TOPIC);
144:                 if (correctTopic != null) {
145:                     backTopic = correctTopic;
146:                 }
147: 
148:                 this.brokerController.getBrokerStatsManager().incSendBackNums(requestHeader.getGroup(), backTopic);
149: 
150:                 response.setCode(ResponseCode.SUCCESS);
151:                 response.setRemark(null);
152: 
153:                 return response;
154:             default:
155:                 break;
156:         }
157: 
158:         response.setCode(ResponseCode.SYSTEM_ERROR);
159:         response.setRemark(putMessageResult.getPutMessageStatus().name());
160:         return response;
161:     }
162: 
163:     response.setCode(ResponseCode.SYSTEM_ERROR);
164:     response.setRemark("putMessageResult is null");
165:     return response;
166: }

Description: When Consumer fails to consume a message, the interface is called to send back the message. Broker stores messages sent back. In this way, the next time Consumer pulls the message, it can be read sequentially from CommitLog and ConstumeQueue.
[x] Because most logic is very similar to Broker's receiving ordinary messages, TODO is marked as unique logic.
Lines 4 to 7: Initialize the response.
Lines 9 to 20: Hook logic.
[x] Lines 22 to 30: Determine whether there is a consumer grouping.
Lines 32 to 37: Check whether Broker has write permission.
Lines 39 to 44: Check if the number of retry queues is greater than 0.
Line 47: Calculate retry topic.
[x] Line 50: Randomly assign queue numbers, depending on retryQueue Nums.
Lines 52 to 56: Compute sysFlag.
Lines 58 to 72: Get TopicConfig. If it does not exist, create it.
[x] Lines 74 to 80: Query messages. If it does not exist, an exception error is returned.
Lines 82 to 86: Set retryTopic to the message e x tension property.
Line 89: Set the message not to wait for storage to complete.
- When the Broker brush mode is synchronized, it will lead to synchronized dropping and can not be submitted in batches. Is there a problem? If you know something, please let me know. (vi).
[x] Lines 91 to 116: Deal with delayLevel.
Lines 118 to 131: Create Message Ext Broker Inner.
Lines 133 to 135: Set the original message number to the e x tended properties.
Lines 137 to 161: Add messages.

7. Ending

Thank you for reading, collecting and commenting on this article.

(ii) If there is a problem with the parsing or if there is a misunderstanding, apologize. If it's convenient, add QQ: 7685413. Let's have a 1:1 exchange.

Thank you again.

Posted by dercof on Tue, 02 Jul 2019 13:41:04 -0700

Programmer Group