this section describes the process of Consumer receiving messages, which is divided into Pull and Push modes.
1. initialization
as mentioned in the previous section, there are two ways for a Consumer to accept a client:
- Broker finds that the client list has changed and informs all consumers to perform Rebalance
- Consumer automatically performs Rebalance every 20 seconds
When the notification of 1. Arrives at the Consumer, the Rebalance will be triggered immediately, and then the timer waiting time of 2. Will be reset. The last way to notify Consumer is
- Push mode: when a new Queue is assigned to the client, a new PullRequest will be wrapped for subsequent automatic pull messages. For details, please refer to the executepullrequestimmediate method of DefaultMQPushConsumerImpl
- Pull mode: the MessageQueueListener calling DefaultMQPullConsumerImpl has a Queue change
2. Push mode
executePullRequestImmediately:
public void executePullRequestImmediately(final PullRequest pullRequest) { this.mQClientFactory.getPullMessageService().executePullRequestImmediately(pullRequest); }
Pass the PullRequest object to the executepullrequestimmediate method of the PullMessageService:
public void executePullRequestImmediately(final PullRequest pullRequest) { try { this.pullRequestQueue.put(pullRequest); } catch (InterruptedException e) { log.error("executePullRequestImmediately pullRequestQueue.put", e); } }
The structure of PullMessageService is as follows:
A LinkedBlockingQueue property pullRequestQueue is maintained internally to store the pending PullRequest, and a ScheduledExecutorService is used to postpone the processing of PullRequest. The specific process is as follows:
- RebalanceImpl calls the executepullrequestimmediate method of DefaultMQPushConsumerImpl, passing in PullRequest
- DefaultMQPushConsumerImpl internally calls the executepullrequestimmediate method of the PullMessageService, which places the incoming PullRequest object in the LinkedBlockingQueue for storage and waits for subsequent processing.
- PullMessageService will loop out a PullRequest from the queue and call its own pullMessage for subsequent processing. This action selects the corresponding client instance DefaultMQPushConsumerImpl from MQClientInstance and delegates it's pullMessage method.
- DefaultMQPushConsumerImpl will first determine whether the current request meets the conditions. If it does not meet the conditions, it will call the executePullRequestLater method of PullMessage to delay the processing of the current request. If the conditions are met, a PullCallback object is encapsulated for processing asynchronous messages, and the PullAPIWrapper asynchronous request Broker is called to pull messages.
From the above process, it can be seen that the Push mode is still pulled by the client actively, that is, the so-called encapsulation pull mode to realize Push mode. The simple diagram is as follows:
Internally, pull data from the corresponding MessageQueue of PullRequest through the PullMessageService loop.
2.1. DefaultMQPushConsumerImpl.pullMessage(PullRequest)
this method is used to pull messages from MessageQueue. The main process is as follows:
-
Judge whether the PullRequest corresponding to the MessageQueue has been marked as drop, and if so, return it directly
-
Carry out a series of checks. If the check fails, wait for a certain time and then put it back into the queue to be processed by PullMessageService, mainly through the ScheduledExecutorService in PullMessageService to delay the execution. The situations involved include:
- If the client is not ready (the status is running after DefaultMQPushCOnsumerImpl start s), delay pull "time" delay "miles" when "exception (3000) and put it back in the queue of PullMessage
- If it is in a suspended state, delay pull "time" delay "miles" when "suspend (1000) and then put it back into the waiting queue of PullMessageService
- If the number of cached messages is greater than the configured pull threads threshold (1000 by default), wait for pull? Time? Delay? Miles? When? Flow? Control (50) before returning to the waiting queue for processing
- If the cached message size is greater than the configured pull size threshold (default 100M), wait for PULL TIME DELAY MILLS WHEN FLOW CONTROL (50) before returning to the wait queue for processing
- If the offset of the difference between the cached data offsets exceeds the set value (2000 by default), wait for pull? Time? Delay? Miles? When? Flow? Control (50) before returning to the waiting queue for processing
- If you do not subscribe to the topic corresponding to MessageQueue, wait for pull "time" delay "miles" when "exception (3000) before returning to the queue for processing
-
Wrap the PullCallback object and call the PullAPIWrapper to initiate an asynchronous request pull message
After receiving the result through the PullAPIWrapper, the above will wrap the result as a PullResult object and call back the PullCallback. PullCallback and PullResult are defined as follows:
public interface PullCallback { void onSuccess(final PullResult pullResult); void onException(final Throwable e); }
public class PullResult { private final PullStatus pullStatus;//Request state private final long nextBeginOffset;//offset returned by Broker for next consumption private final long minOffset; private final long maxOffset; private List<MessageExt> msgFoundList;//Message list, return one batch of messages at a time }
The following is the flow of the pullMessage method to process asynchronous return results:
- If the request fails, wait for pull "time" delay "miles" when "exception (3000) and put it back into the queue to be processed by PullMessageService; if the processing is successful, enter 2
- Call the PullAPIWrapper to preprocess the result
- Process according to request status
- New news (FOUND)
- Set the start position of next consumption of PullRequest to nextBeginOffset of PullResult
- If the result list is empty, it will not be delayed, and it will be put into the waiting queue of PullMessageService immediately, otherwise it will enter 3
- Put the result list < messageext > in PullResult into the cache of ProcessQueue, and inform ConsumeMessageService to process
- Put the PullRequest back to the waiting queue for further processing. If there is a set pull interval, wait for that time and then turn it to the queue for processing. Otherwise, put it directly in the queue for processing
- No new MSG
- Set the start position of next consumption of PullRequest to nextBeginOffset of PullResult
- Update the offset store if the number of messages to be consumed in the cache is 0
- Put PullRequest into the pending queue of PullMessageService immediately
- No matched MSG
- Set the start position of next consumption of PullRequest to nextBeginOffset of PullResult
- Update the offset store if the number of messages to be consumed in the cache is 0
- Put PullRequest into the pending queue of PullMessageService immediately
- Illegal offset (offset? Illegal)
- Set the start position of next consumption of PullRequest to nextBeginOffset of PullResult
- Mark the pullrequse as drop
- Update and persist the consumption offset after 10s, and then notify Rebalance to remove the MessageQueue
- New news (FOUND)
next, we will introduce ProcessQueue, which only identifies several related properties:
public class ProcessQueue { private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock(); //Cached messages to be consumed, sorted by the initial offset of the message private final TreeMap</*Message start offset*/Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>(); //Number of messages to be consumed cached private final AtomicLong msgCount = new AtomicLong(); //Size of cached messages to be consumed private final AtomicLong msgSize = new AtomicLong(); private final Lock lockConsume = new ReentrantLock(); /** * A subset of msgTreeMap, will only be used when orderly consume */ private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>(); private final AtomicLong tryUnlockTimes = new AtomicLong(0); private volatile long queueOffsetMax = 0L; private volatile boolean dropped = false; //Last pull time private volatile long lastPullTimestamp = System.currentTimeMillis(); //Time recently consumed by client private volatile long lastConsumeTimestamp = System.currentTimeMillis(); private volatile boolean locked = false; private volatile long lastLockTimestamp = System.currentTimeMillis(); //Whether it is currently in consumption, which is used for sequential consumption mode, and is not valid for parallel consumption private volatile boolean consuming = false; private volatile long msgAccCnt = 0; }
ProcessQueue shows the consumption of MessageQueue. As mentioned above, if there is any data after the pull request is initiated, it will be put into the cache of ProcessQueue, that is, the msgTreeMap attribute, so the cached messages will be sorted and stored according to the initial offset of the message. The current processing status of MessageQueue can be viewed through ProcessQueue, which is also used to assist in the realization of sequential consumption.
2.2 ConsumeMessageService
the message content returned asynchronously will be handed over to consummessageservice, which is an interface. The method definition is as follows:
public interface ConsumeMessageService { void start(); void shutdown(); void updateCorePoolSize(int corePoolSize); void incCorePoolSize(); void decCorePoolSize(); int getCorePoolSize(); ConsumeMessageDirectlyResult consumeMessageDirectly(final MessageExt msg, final String brokerName); void submitConsumeRequest( final List<MessageExt> msgs, final ProcessQueue processQueue, final MessageQueue messageQueue, final boolean dispathToConsume); }
It can be seen from the definition that the implementation class is required to provide the function of asynchronous processing. The implementation classes provided internally are:
Consummessageconcurrencyservice: parallel consumption; consummessageorderlyservice: sequential consumption. Here we focus on consummessageconcurrencyservice. After asynchronous request, the pulled new message list will be submitted to the submitConsumeRequest method for processing, as follows:
This method will split the incoming message list into a ConsumeRequest and mention it to the thread pool for processing. If the length of the incoming message list exceeds the set value (1 by default), multiple batches will be processed.
before introducing the specific process of consumption, review the Demo of the client startup process, and write the received message as follows:
public class Consumer { public static void main (String[] args) throws InterruptedException, MQClientException { // Instantiate consumer DefaultMQPushConsumer consumer = new DefaultMQPushConsumer ("GroupTest"); // Set the address of NameServer consumer.setNamesrvAddr ("localhost:9876"); // Subscribe to one or more topics and tags to filter the messages to be consumed consumer.subscribe ("TopicTest", "*"); // Register the callback implementation class to handle the messages pulled back from the broker consumer.registerMessageListener (new MessageListenerConcurrently () { @Override public ConsumeConcurrentlyStatus consumeMessage (List<MessageExt> msgs, ConsumeConcurrentlyContext context) { System.out.printf ("%s Receive New Messages: %s %n", Thread.currentThread ().getName (), msgs); // Mark that the message has been successfully consumed return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } }); // Start consumer instance consumer.start (); System.out.printf ("Consumer Started.%n"); } }
A messagelistenercurrently is registered, which will be used by the client to process messages.
looking back at ConsumeRequest, this class implements the Runnable interface and will complete the main processing work in the run method. The main actions are:
- Call DefaultMQPushConsumerImpl.executeHookBefore to perform the pre hook action
- Call messagelistenercurrently.consumemessage to inform the client to process the message, that is, the above demo content, and the result ConsumeConcurrentlyStatus will be returned
- Call DefaultMQPushConsumerImpl.executeHookAfter to perform the post hook action
- ConsumeMessageConcurrentlyService.processConsumeResult perform the closing action according to consumercurrentlystatus
2.2.1. MessageListenerConcurrently.consumeMessage
where the user actually receives the message and performs the processing action, it needs to return ConsumeConcurrentlyStatus to inform the framework of the processing result. In this method, it is better not to do time-consuming tasks, and return the results to the framework after quick processing to avoid message accumulation in the process pool. The message content can be copied once and then put into the thread pool for distribution.
2.2.2. ConsumeMessageConcurrentlyService.processConsumeResult
this method mainly performs the closing action after the user has consumed the data. The process is as follows:
ConsumerRequest instantiates a consumercurrentlycontext object at the beginning of the run method for subsequent delivery of content. The definition is:
public class ConsumeConcurrentlyContext { private final MessageQueue messageQueue; //Delay level of retry, - 1: no retry; 0: controlled by broker; > 0 controlled by client private int delayLevelWhenNextConsume = 0; //Message index number of the last normal consumption in the message list private int ackIndex = Integer.MAX_VALUE; }
Where ackIndex indicates the index number of the last normal consumption message (0 from the beginning, 0~ackIndex is normal consumption). The message after this position indicates that normal consumption is not possible. This value is controlled by the client, and the message to be resend can be controlled through ackIndex.
The default value of ackIndex is integer.max ᜇ value. If this value is set, all messages are considered to be consumed normally and there is no error. In the above process, the success and failure are also judged according to the ackIndex. For the messages after the ackIndex, if they are in the cluster consumption mode, they will be sent back to the broker first, and the broker will control the retry time. If the retry fails, these failed messages will be collected, with a delay of 5 seconds, and then the ConsumeMessageService.submitConsumeRequest will be called again to let the client process again. Finally, the successful message will be removed from ProcessQueue, the cache will be updated, and then the offset of q consumption will be recorded, waiting for the background thread to synchronize to broker or local.
based on the above introduction, the processing flow in Push mode is roughly as follows:
Push mode pulls messages from the monitored MessageQueue in Pull mode through the PullMessageService loop and distributes them to the registered MesageListenerConsurrently object of the user for processing. After that, it will automatically handle the retry, offset update and other actions of the message, so as to simulate the active push of the message from the Broker side.
2. Pull mode
like Push mode, the Pull mode is triggered through Rebalance, as follows:
As mentioned at the beginning, there is a Queue change in the MessageQueueListener that will call back DefaultMQPullConsumerImpl.
the system provides MQPullConsumerScheduleService, which can Pull messages in Pull mode regularly and notify MessageQueueListener of the results. The internal implementation is as follows:
class MessageQueueListenerImpl implements MessageQueueListener { @Override public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {//mqAll all q under the topic, mqDivided q assigned to the instance MessageModel messageModel = MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel(); switch (messageModel) { case BROADCASTING: MQPullConsumerScheduleService.this.putTask(topic, mqAll);//Notify the listener under this topic of all the latest q break; case CLUSTERING: MQPullConsumerScheduleService.this.putTask(topic, mqDivided);//Notify the listener under the topic, and the q assigned by the instance break; default: break; } } }
putTask will package the new MessageQueue assigned to as a PullTaskImpl. PullTaskImpl implements Runnable and will be executed all the time in the background. Instead, it will stop the PullTaskImpl corresponding to the MessageQueue that does not belong to its own processing. PullTaskImpl will find the processing class PullTaskCallback corresponding to the topic the MessageQueue listens to, call doPullTask, and let the user handle the specific action.
The example of MQPullConsumerScheduleService is:
public class PullScheduleService { public static void main(String[] args) throws MQClientException { final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService("GroupName1"); scheduleService.setMessageModel(MessageModel.CLUSTERING); scheduleService.registerPullTaskCallback("TopicTest", new PullTaskCallback() {//Register listeners for topic @Override public void doPullTask(MessageQueue mq, PullTaskContext context) { MQPullConsumer consumer = context.getPullConsumer(); try { long offset = consumer.fetchConsumeOffset(mq, false); if (offset < 0) offset = 0; PullResult pullResult = consumer.pull(mq, "*", offset, 32); System.out.printf("%s%n", offset + "\t" + mq + "\t" + pullResult); switch (pullResult.getPullStatus()) { case FOUND: break; case NO_MATCHED_MSG: break; case NO_NEW_MSG: case OFFSET_ILLEGAL: break; default: break; } consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());//Report the offset of consumption, and actively report after consumption context.setPullNextDelayTimeMillis(100);//Set next trigger interval } catch (Exception e) { e.printStackTrace(); } } }); scheduleService.start(); } }
you can also manually execute pull by yourself, as shown in the following example:
public class PullConsumer { private static final Map<MessageQueue, Long> OFFSE_TABLE = new HashMap<MessageQueue, Long>(); public static void main(String[] args) throws MQClientException { DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_5"); consumer.start(); Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("TopicTest1"); for (MessageQueue mq : mqs) { System.out.printf("Consume from the queue: %s%n", mq); SINGLE_MQ: while (true) { try { PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32); System.out.printf("%s%n", pullResult); putMessageQueueOffset(mq, pullResult.getNextBeginOffset()); switch (pullResult.getPullStatus()) { case FOUND: break; case NO_MATCHED_MSG: break; case NO_NEW_MSG: break SINGLE_MQ; case OFFSET_ILLEGAL: break; default: break; } } catch (Exception e) { e.printStackTrace(); } } } consumer.shutdown(); } private static long getMessageQueueOffset(MessageQueue mq) { Long offset = OFFSE_TABLE.get(mq); if (offset != null) return offset; return 0; } private static void putMessageQueueOffset(MessageQueue mq, long offset) { OFFSE_TABLE.put(mq, offset); } }
compared with Push mode, Pull mode requires users to control message retry, offset update and other actions. Here is a sketch of the notes made during the source reading process of this part:
More original content, please search WeChat official account: doubaotaizi