With the gradual decomposition of services in micro-service architecture, database privatization has become a consensus, which also leads to the problem of distributed transactions facing micro-service landing process as a very difficult obstacle to overcome, but there is no complete universal solution. In order to ensure the consistency of distributed transactions, there are two mature solutions in the industry, namely, two-stage commit protocol (2PC), three-stage commit protocol (3PC), TCC proposed by Ali, etc. RocketMQ adopts 2PC (two-stage protocol) and compensation mechanism (transaction review). RocketMQ started supporting transactions in version 4.3.0.
1. Overview
RocketMQ transaction message design is mainly to solve the atomicity of message sending and local transaction execution on the Producer side. RocketMQ's two-way communication capability between broker and producer makes broker inherently exist as a transaction coordinator; RocketMQ itself provides storage mechanism. The High Availability Mechanism of RocketMQ and reliable message design provide persistence capability for transaction messages, which can still ensure the final consistency of transactions in case of system abnormalities.
2. Code Implementation
2.1,producer
public class TransactionProducer { public static void main(String[] args) throws MQClientException, InterruptedException { TransactionListener transactionListener = new TransactionListenerImpl(); TransactionMQProducer producer = new TransactionMQProducer("please_rename_unique_group_name"); ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory() { @Override public Thread newThread(Runnable r) { Thread thread = new Thread(r); thread.setName("client-transaction-msg-check-thread"); return thread; } }); producer.setNamesrvAddr("10.10.15.246:9876;10.10.15.247:9876"); producer.setExecutorService(executorService); producer.setTransactionListener(transactionListener); producer.start(); String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"}; for (int i = 0; i < 10; i++) { try { Message msg = new Message("TranTest", tags[i % tags.length], "KEY" + i, ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)); SendResult sendResult = producer.sendMessageInTransaction(msg, null); System.out.printf("%s%n", sendResult); Thread.sleep(10); } catch (MQClientException | UnsupportedEncodingException e) { e.printStackTrace(); } } for (int i = 0; i < 100000; i++) { Thread.sleep(1000); } producer.shutdown(); } }
Transaction listener
public class TransactionListenerImpl implements TransactionListener { private AtomicInteger transactionIndex = new AtomicInteger(0); private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>(); public LocalTransactionState executeLocalTransaction(Message msg, Object arg) { int value = transactionIndex.getAndIncrement(); int status = value % 3; localTrans.put(msg.getTransactionId(), status); return LocalTransactionState.UNKNOW; } public LocalTransactionState checkLocalTransaction(MessageExt msg) { Integer status = localTrans.get(msg.getTransactionId()); if (null != status) { switch (status) { case 0: return LocalTransactionState.UNKNOW; case 1: return LocalTransactionState.COMMIT_MESSAGE; case 2: return LocalTransactionState.ROLLBACK_MESSAGE; } } return LocalTransactionState.COMMIT_MESSAGE; } }
Send 10 messages, divide by 3, 0: state is unknown, 1: submit message, 2: rollback message
2.2,consumer
public class TransactionConsumer { public static void main(String[] args){ try { DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(); consumer.setConsumerGroup("consumer_test_clustering"); consumer.setNamesrvAddr("10.10.15.246:9876;10.10.15.247:9876"); consumer.subscribe("TranTest", "*"); consumer.registerMessageListener(new MessageListenerConcurrently(){ @Override public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> paramList, ConsumeConcurrentlyContext paramConsumeConcurrentlyContext) { try { for(MessageExt msg : paramList){ String msgbody = new String(msg.getBody(), "utf-8"); System.out.println("Consumer=== MessageBody: "+ msgbody);//Output message content } } catch (Exception e) { e.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; //try again later } return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; //Consumption Success } }); consumer.start(); System.out.println("Consumer===Successful start-up!"); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } } }
View the results:
It's the same as our expectation. The balance of less than 10 divided by 3 is only 1/4/7.
3. Internal mechanism
As an asynchronous assured transaction, transaction message decouples two transaction branches asynchronously through MQ. The design process of RocketMQ transaction message also draws on the two-stage commit theory. The overall interaction process is shown in the following figure:
- The transaction initiator first sends a prepare message to MQ.
- Execute the local transaction after sending the prepare message successfully.
- Return commit or rollback based on local transaction execution results.
- If the message is rollback, MQ will delete the prepare message and not send it down. If it is a commit message, MQ will send the message to the consumer.
- If the execution end is suspended or timed out during the execution of a local transaction, MQ will constantly ask other producer s in its group for status.
- Consumer-side consumption success mechanism has MQ guarantee.
4. Source code analysis
4.1. Transaction message sending
View the flow chart of message sending
Transaction MQ Producer Transaction Message Producer
First, the differences between Transaction MQ Producer and ordinary producers are analyzed.
public class TransactionMQProducer extends DefaultMQProducer { private int checkThreadPoolMinSize = 1;//Core thread pool size private int checkThreadPoolMaxSize = 1;//Maximum number of threads private int checkRequestHoldMax = 2000;//Thread Waiting Queue //Transaction status review asynchronous execution thread pool private ExecutorService executorService; //Transaction listener to implement local transaction status execution results and local transaction status review private TransactionListener transactionListener; public TransactionMQProducer() { } public TransactionMQProducer(final String producerGroup) { this(null, producerGroup, null); } public TransactionMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook) { super(namespace, producerGroup, rpcHook); } @Override public void start() throws MQClientException { //At startup, there is one more way to initialize the transaction environment than normal messages this.defaultMQProducerImpl.initTransactionEnv(); super.start(); } @Override public void shutdown() { super.shutdown(); //When you close, you also need to close the transaction environment this.defaultMQProducerImpl.destroyTransactionEnv(); } @Overrideu public TransactionSendResult sendMessageInTransaction(final Message msg, final Object arg) throws MQClientException { //Transaction message sending must implement transaction listener interface amount to query local transaction status and transaction review if (null == this.transactionListener) { throw new MQClientException("TransactionListener is null", null); } return this.defaultMQProducerImpl.sendMessageInTransaction(msg, null, arg); } }
1.1. ExecutorService transaction review thread pool
Transaction status review asynchronous execution thread pool, there is no method to initialize thread pool, we found that there is a method to initialize transaction environment at startup, initTransaction Env ()
public void initTransactionEnv() { TransactionMQProducer producer = (TransactionMQProducer) this.defaultMQProducer; if (producer.getExecutorService() != null) { this.checkExecutor = producer.getExecutorService(); } else { this.checkRequestQueue = new LinkedBlockingQueue<Runnable>(producer.getCheckRequestHoldMax()); this.checkExecutor = new ThreadPoolExecutor( producer.getCheckThreadPoolMinSize(), producer.getCheckThreadPoolMaxSize(), 1000 * 60, TimeUnit.MILLISECONDS, this.checkRequestQueue); } }
You can set some parameters of thread pool by yourself. If not, the default thread number of thread pool is "1", the maximum idle time of thread is 60 seconds, and the thread waits for queue 2000.
TransactionListener transaction listener
public interface TransactionListener { LocalTransactionState executeLocalTransaction(final Message msg, final Object arg); LocalTransactionState checkLocalTransaction(final MessageExt msg); }
executeLocalTransaction: Executing local transactions, our own business logic code
CheckLocal Transaction: Transaction message status review
2. Transaction message sending
public TransactionSendResult sendMessageInTransaction(final Message msg, final LocalTransactionExecuter localTransactionExecuter, final Object arg) throws MQClientException { TransactionListener transactionListener = getCheckListener(); if (null == localTransactionExecuter && null == transactionListener) { throw new MQClientException("tranExecutor is null", null); } Validators.checkMessage(msg, this.defaultMQProducer); SendResult sendResult = null; //Transaction attribute TRAN_MSG=true for adding messages MessageAccessor.putProperty(msg, MessageConst.PROPERTY_TRANSACTION_PREPARED, "true"); //Add the transaction attribute PGROUP of the message as the producer group name MessageAccessor.putProperty(msg, MessageConst.PROPERTY_PRODUCER_GROUP, this.defaultMQProducer.getProducerGroup()); try { //Core code for sending messages sendResult = this.send(msg); } catch (Exception e) { throw new MQClientException("send message Exception", e); } LocalTransactionState localTransactionState = LocalTransactionState.UNKNOW; Throwable localException = null; switch (sendResult.getSendStatus()) { case SEND_OK: { try { if (sendResult.getTransactionId() != null) { //Setting the properties of messages _transactionId__ msg.putUserProperty("__transactionId__", sendResult.getTransactionId()); } String transactionId = msg.getProperty(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX); if (null != transactionId && !"".equals(transactionId)) { msg.setTransactionId(transactionId); } if (null != localTransactionExecuter) { localTransactionState = localTransactionExecuter.executeLocalTransactionBranch(msg, arg); } else if (transactionListener != null) { log.debug("Used new transaction API"); //Executing local transactions to obtain local transaction status localTransactionState = transactionListener.executeLocalTransaction(msg, arg); } if (null == localTransactionState) { localTransactionState = LocalTransactionState.UNKNOW; } // if (localTransactionState != LocalTransactionState.COMMIT_MESSAGE) { log.info("executeLocalTransactionBranch return {}", localTransactionState); log.info(msg.toString()); } } catch (Throwable e) { log.info("executeLocalTransactionBranch exception", e); log.info(msg.toString()); localException = e; } } break; case FLUSH_DISK_TIMEOUT: case FLUSH_SLAVE_TIMEOUT: case SLAVE_NOT_AVAILABLE: localTransactionState = LocalTransactionState.ROLLBACK_MESSAGE; break; default: break; } try { //Status of messages sent to Broker transactions this.endTransaction(sendResult, localTransactionState, localException); } catch (Exception e) { log.warn("local transaction execute " + localTransactionState + ", but end broker transaction failed", e); } TransactionSendResult transactionSendResult = new TransactionSendResult(); transactionSendResult.setSendStatus(sendResult.getSendStatus()); transactionSendResult.setMessageQueue(sendResult.getMessageQueue()); transactionSendResult.setMsgId(sendResult.getMsgId()); transactionSendResult.setQueueOffset(sendResult.getQueueOffset()); transactionSendResult.setTransactionId(sendResult.getTransactionId()); transactionSendResult.setLocalTransactionState(localTransactionState); return transactionSendResult; }
Analysis source code
Lines 11-15: Transaction message adds attribute TRAN_MSG=true, and PGROUP is the producer group name, representing the producer group of the transaction prepare message and the message to which the message belongs. When a producer group is set up to review the local transaction status for transaction messages, a producer can be randomly selected from the producer group.
Line 18: Core code for sending messages, highlighted below
Lines 26-57: prepare message sent successfully, transactionListener.executeLocalTransaction executed by business code to get its transaction status
Line 69: broker sends the end of the transaction and the status of the transaction.
- LocalTransactionState.COMMIT_MESSAGE: Submitting transactions
- LocalTransactionState.ROLLBACK_MESSAGE: Rollback transactions
- Local Transaction State. UNKNOW: Unknown transaction state
2.1. DefaultMQProducerImpl.sendKernelImpl, the core method of sending messages
The method is too long. We extract the transaction setup code for analysis.
final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED); if (tranMsg != null && Boolean.parseBoolean(tranMsg)) { sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE; }
Get the attribute TRAN_MSG of the message, which we set earlier. The tag message is TRANSACTION_PREPARED_TYPE
SendMessageRequestHeader requestHeader = new SendMessageRequestHeader(); requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup()); requestHeader.setTopic(msg.getTopic()); requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey()); requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums()); requestHeader.setQueueId(mq.getQueueId()); requestHeader.setSysFlag(sysFlag); requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
Set the SysFlag attribute TRANSACTION_PREPARED_TYPE in the request header sent to the broker message, and the request header sets Properties as the attribute of the message.
2.2. Broker receives messages
private RemotingCommand sendMessage(final ChannelHandlerContext ctx, final RemotingCommand request, final SendMessageContext sendMessageContext, final SendMessageRequestHeader requestHeader) throws RemotingCommandException { //Eliminate other code... Map<String, String> oriProps = MessageDecoder.string2messageProperties(requestHeader.getProperties()); String traFlag = oriProps.get(MessageConst.PROPERTY_TRANSACTION_PREPARED); if (traFlag != null && Boolean.parseBoolean(traFlag)) { if (this.brokerController.getBrokerConfig().isRejectTransactionMessage()) { response.setCode(ResponseCode.NO_PERMISSION); response.setRemark( "the broker[" + this.brokerController.getBrokerConfig().getBrokerIP1() + "] sending transaction message is forbidden"); return response; } putMessageResult = this.brokerController.getTransactionalMessageService().prepareMessage(msgInner); } else { putMessageResult = this.brokerController.getMessageStore().putMessage(msgInner); } return handlePutMessageResult(putMessageResult, response, request, msgInner, responseHeader, sendMessageContext, ctx, queueIdInt); }
Analysis source code
Lines 7-9: Get the message attributes in the request header, get TRAN_MSG, and determine whether it is a transaction message. According to the tag, the logic of transaction storage and common message is separately used.
Line 17: prepareMessage, the core method of transactional message storage
2.3. Broker's Processing before Storing prepare Messages
TransactionalMessageService Impl. prepareMessage (); Call TransactionalMessageBridge.putHalfMessage()
public PutMessageResult putHalfMessage(MessageExtBrokerInner messageInner) { return store.putMessage(parseHalfMessageInner(messageInner)); } private MessageExtBrokerInner parseHalfMessageInner(MessageExtBrokerInner msgInner) { //Store the original message topic in the properties MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_TOPIC, msgInner.getTopic()); //The ID of the original message queue is stored in the attribute MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msgInner.getQueueId())); msgInner.setSysFlag( MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), MessageSysFlag.TRANSACTION_NOT_TYPE)); //Set Topic=RMQ_SYS_TRANS_HALF_TOPIC msgInner.setTopic(TransactionalMessageUtil.buildHalfTopic()); //Set the ID of the queue = 0 msgInner.setQueueId(0); msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgInner.getProperties())); return msgInner; }
Some processing of parseHalfMessageInner () was done before storing the message.
13-16: Set topic Topic=RMQ_SYS_TRANS_HALF_TOPIC and queue ID=0
Here we can see that the storage is not the original topic, but the new topic information and queue information, which is somewhat similar to the delayed message.
3. Get the result of sending and execute the final transaction method
DefaultMQProducerImpl.endTransaction() is sent to the broker's final transaction method, encapsulating the request parameters.
public void endTransaction( final SendResult sendResult, final LocalTransactionState localTransactionState, final Throwable localException) throws RemotingException, MQBrokerException, InterruptedException, UnknownHostException { final MessageId id; if (sendResult.getOffsetMsgId() != null) { id = MessageDecoder.decodeMessageId(sendResult.getOffsetMsgId()); } else { id = MessageDecoder.decodeMessageId(sendResult.getMsgId()); } String transactionId = sendResult.getTransactionId(); final String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(sendResult.getMessageQueue().getBrokerName()); EndTransactionRequestHeader requestHeader = new EndTransactionRequestHeader(); requestHeader.setTransactionId(transactionId); requestHeader.setCommitLogOffset(id.getOffset()); switch (localTransactionState) { case COMMIT_MESSAGE: requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_COMMIT_TYPE); break; case ROLLBACK_MESSAGE: requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_ROLLBACK_TYPE); break; case UNKNOW: requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_NOT_TYPE); break; default: break; } requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup()); requestHeader.setTranStateTableOffset(sendResult.getQueueOffset()); requestHeader.setMsgId(sendResult.getMsgId()); String remark = localException != null ? ("executeLocalTransactionBranch exception: " + localException.toString()) : null; this.mQClientFactory.getMQClientAPIImpl().endTransactionOneway(brokerAddr, requestHeader, remark, this.defaultMQProducer.getSendMsgTimeout()); }
We found that the local TransactionState state was encapsulated in requestHeader.setCommitOrRollback().
Local Transaction State and Message SysFlag Correspondence
- LocalTransactionState.COMMIT_MESSAGE == MessageSysFlag.TRANSACTION_COMMIT_TYPE
- LocalTransactionState.ROLLBACK_MESSAGE == MessageSysFlag.TRANSACTION_ROLLBACK_TYPE
- LocalTransactionState.UNKNOW == MessageSysFlag.TRANSACTION_NOT_TYPE
3.1. Broker's approach to the final transaction
Analysis of EndTransaction Processor. ProceRequest () Processing Method
public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request) throws RemotingCommandException { //Eliminate other code... if (MessageSysFlag.TRANSACTION_COMMIT_TYPE == requestHeader.getCommitOrRollback()) { result = this.brokerController.getTransactionalMessageService().commitMessage(requestHeader); if (result.getResponseCode() == ResponseCode.SUCCESS) { RemotingCommand res = checkPrepareMessage(result.getPrepareMessage(), requestHeader); if (res.getCode() == ResponseCode.SUCCESS) { MessageExtBrokerInner msgInner = endMessageTransaction(result.getPrepareMessage()); msgInner.setSysFlag(MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), requestHeader.getCommitOrRollback())); msgInner.setQueueOffset(requestHeader.getTranStateTableOffset()); msgInner.setPreparedTransactionOffset(requestHeader.getCommitLogOffset()); msgInner.setStoreTimestamp(result.getPrepareMessage().getStoreTimestamp()); RemotingCommand sendResult = sendFinalMessage(msgInner); if (sendResult.getCode() == ResponseCode.SUCCESS) { this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage()); } return sendResult; } return res; } } else if (MessageSysFlag.TRANSACTION_ROLLBACK_TYPE == requestHeader.getCommitOrRollback()) { result = this.brokerController.getTransactionalMessageService().rollbackMessage(requestHeader); if (result.getResponseCode() == ResponseCode.SUCCESS) { RemotingCommand res = checkPrepareMessage(result.getPrepareMessage(), requestHeader); if (res.getCode() == ResponseCode.SUCCESS) { this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage()); } return res; } } response.setCode(result.getResponseCode()); response.setRemark(result.getResponseRemark()); return response; }
Lines 4-22: Transaction messages are business processes of the type submitted, as analyzed above by COMMIT_MESSAGE=== TRANSACTION_COMMIT_TYPE
Lines 24-35: Business processing for transaction rollback, ROLLBACK_MESSAGE = TRANSACTION_ROLLBACK_TYPE
3.2. Submit Transaction Processing Flow
Core Code Analysis of Submitting Transactions
MessageExtBrokerInner msgInner = endMessageTransaction(result.getPrepareMessage()); msgInner.setSysFlag(MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), requestHeader.getCommitOrRollback())); msgInner.setQueueOffset(requestHeader.getTranStateTableOffset()); msgInner.setPreparedTransactionOffset(requestHeader.getCommitLogOffset()); msgInner.setStoreTimestamp(result.getPrepareMessage().getStoreTimestamp()); RemotingCommand sendResult = sendFinalMessage(msgInner); if (sendResult.getCode() == ResponseCode.SUCCESS) { this.brokerController.getTransactionalMessageService().deletePrepareMessage(result.getPrepareMessage()); }
Line 1: It is the processing of messages, endMessage Transaction, that sets the relevant message parameters
Line 6: sendFinalMessage sends the final message
Line 8: deletePrepareMessage deletes prepare message
- Let's first analyze endMessage Transaction to assemble message data
private MessageExtBrokerInner endMessageTransaction(MessageExt msgExt) { MessageExtBrokerInner msgInner = new MessageExtBrokerInner(); //Setting the original topic of the message is retrieved from the properties of the message, set when the message is sent, as analyzed earlier msgInner.setTopic(msgExt.getUserProperty(MessageConst.PROPERTY_REAL_TOPIC)); //The ID of the original queue setting the message is obtained from the properties of the message, which is set when the message is sent, as analyzed earlier. msgInner.setQueueId(Integer.parseInt(msgExt.getUserProperty(MessageConst.PROPERTY_REAL_QUEUE_ID))); msgInner.setBody(msgExt.getBody()); msgInner.setFlag(msgExt.getFlag()); msgInner.setBornTimestamp(msgExt.getBornTimestamp()); msgInner.setBornHost(msgExt.getBornHost()); msgInner.setStoreHost(msgExt.getStoreHost()); msgInner.setReconsumeTimes(msgExt.getReconsumeTimes()); msgInner.setWaitStoreMsgOK(false); msgInner.setTransactionId(msgExt.getUserProperty(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX)); msgInner.setSysFlag(msgExt.getSysFlag()); TopicFilterType topicFilterType = (msgInner.getSysFlag() & MessageSysFlag.MULTI_TAGS_FLAG) == MessageSysFlag.MULTI_TAGS_FLAG ? TopicFilterType.MULTI_TAG : TopicFilterType.SINGLE_TAG; long tagsCodeValue = MessageExtBrokerInner.tagsString2tagsCode(topicFilterType, msgInner.getTags()); msgInner.setTagsCode(tagsCodeValue); MessageAccessor.setProperties(msgInner, msgExt.getProperties()); msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgExt.getProperties())); MessageAccessor.clearProperty(msgInner, MessageConst.PROPERTY_REAL_TOPIC); MessageAccessor.clearProperty(msgInner, MessageConst.PROPERTY_REAL_QUEUE_ID); return msgInner; }
Lines 2-4: Set the original topic Topic of the message and the ID of the message queue
-
sendFinalMessage does not need to be analyzed to send assembled messages to the corresponding topics and queues for consumers to use, just like regular messages.
-
deletePrepareMessage Deletes the prepare Message
The core method of calling TransactionalMessageBridge.putOpMessage
public boolean putOpMessage(MessageExt messageExt, String opType) { MessageQueue messageQueue = new MessageQueue(messageExt.getTopic(), this.brokerController.getBrokerConfig().getBrokerName(), messageExt.getQueueId()); if (TransactionalMessageUtil.REMOVETAG.equals(opType)) { return addRemoveTagInTransactionOp(messageExt, messageQueue); } return true; }
Line 5: Call the add transaction message operation and analyze its source code
private boolean addRemoveTagInTransactionOp(MessageExt messageExt, MessageQueue messageQueue) { //Message topic=RMQ_SYS_TRANS_OP_HALF_TOPIC Message message = new Message(TransactionalMessageUtil.buildOpTopic(), TransactionalMessageUtil.REMOVETAG, String.valueOf(messageExt.getQueueOffset()).getBytes(TransactionalMessageUtil.charset)); writeOp(message, messageQueue); return true; }
Line 3: We find that the topic of the created message is RMQ_SYS_TRANS_OP_HALF_TOPIC
Five lines: Write the message
We find that deleting preprepremessage is not really deleting, but storing preprepremessage into RMQ_SYS_TRANS_OP_HALF_TOPIC topic to express the transaction message and process it (including submission and rollback), which provides a basis for querying the unprocessed transaction message.
3.3. Rollback Transaction Processing Flow
The process of deletePrepareMessage deleting prepare Message has been analyzed in the commit transaction process, which is not explained here.
4.2. Business Timing Review
View the flow chart for transaction review
The two-stage protocol sends and submits rollback messages. When the status of the local transaction message is UNKNOW, the end of the transaction does not do anything. The transaction status of the sender is rollback or commit by periodic review of the transaction status. The transaction status of the message is reviewed by opening the thread.
public class TransactionalMessageCheckService extends ServiceThread { private BrokerController brokerController; public TransactionalMessageCheckService(BrokerController brokerController) { this.brokerController = brokerController; } @Override public String getServiceName() { return TransactionalMessageCheckService.class.getSimpleName(); } @Override public void run() { // Start transaction check service thread! long checkInterval = brokerController.getBrokerConfig().getTransactionCheckInterval(); while (!this.isStopped()) { this.waitForRunning(checkInterval); } } @Override protected void onWaitEnd() { long timeout = brokerController.getBrokerConfig().getTransactionTimeOut(); int checkMax = brokerController.getBrokerConfig().getTransactionCheckMax(); long begin = System.currentTimeMillis(); this.brokerController.getTransactionalMessageService().check(timeout, checkMax, this.brokerController.getTransactionalMessageCheckListener()); } }
run() opens the thread to do a message status review of the message's transaction by default of 60 seconds. Transaction CheckInterval unit milliseconds can be configured in the broker configuration file. The actual business processing method is to call onWaitEnd(). Call the core method TransactionalMessageServiceImpl.check() core transaction review mechanism method.
public void check(long transactionTimeout, int transactionCheckMax, AbstractTransactionalMessageCheckListener listener) { try { String topic = MixAll.RMQ_SYS_TRANS_HALF_TOPIC; Set<MessageQueue> msgQueues = transactionalMessageBridge.fetchMessageQueues(topic); if (msgQueues == null || msgQueues.size() == 0) { log.warn("The queue of topic is empty :" + topic); return; } log.debug("Check topic={}, queues={}", topic, msgQueues); //Traversing through each message queue for (MessageQueue messageQueue : msgQueues) { long startTime = System.currentTimeMillis(); //Get the RMQ_SYS_TRANS_OP_HALF_TOPIC queue based on the message queue, that is, the transaction message that has been rolled back or commit MessageQueue opQueue = getOpQueue(messageQueue); //Get the current progress of the RMQ_SYS_TRANS_HALF_TOPIC queue long halfOffset = transactionalMessageBridge.fetchConsumeOffset(messageQueue); //Get the current progress of the RMQ_SYS_TRANS_OP_HALF_TOPIC queue long opOffset = transactionalMessageBridge.fetchConsumeOffset(opQueue); log.info("Before check, the queue={} msgOffset={} opOffset={}", messageQueue, halfOffset, opOffset); if (halfOffset < 0 || opOffset < 0) { log.error("MessageQueue: {} illegal offset read: {}, op offset: {},skip this queue", messageQueue, halfOffset, opOffset); continue; } List<Long> doneOpOffset = new ArrayList<>(); HashMap<Long, Long> removeMap = new HashMap<>(); //Find out if a commit or rollback message is stored in the removeMap in the record that will verify whether a check amount is required PullResult pullResult = fillOpRemoveMap(removeMap, opQueue, opOffset, halfOffset, doneOpOffset); if (null == pullResult) { log.error("The queue={} check msgOffset={} with opOffset={} failed, pullResult is null", messageQueue, halfOffset, opOffset); continue; } // single thread int getMessageNullCount = 1; long newOffset = halfOffset; long i = halfOffset; while (true) { //Processing time 60 seconds if (System.currentTimeMillis() - startTime > MAX_PROCESS_TIME_LIMIT) { log.info("Queue={} process time reach max={}", messageQueue, MAX_PROCESS_TIME_LIMIT); break; } //If you have commit or rollback, continue looking for the next one if (removeMap.containsKey(i)) { log.info("Half offset {} has been committed/rolled back", i); removeMap.remove(i); } else { //Get the current transaction message GetResult getResult = getHalfMsg(messageQueue, i); MessageExt msgExt = getResult.getMsg(); if (msgExt == null) { if (getMessageNullCount++ > MAX_RETRY_COUNT_WHEN_HALF_NULL) { break; } if (getResult.getPullResult().getPullStatus() == PullStatus.NO_NEW_MSG) { log.debug("No new msg, the miss offset={} in={}, continue check={}, pull result={}", i, messageQueue, getMessageNullCount, getResult.getPullResult()); break; } else { log.info("Illegal offset, the miss offset={} in={}, continue check={}, pull result={}", i, messageQueue, getMessageNullCount, getResult.getPullResult()); i = getResult.getPullResult().getNextBeginOffset(); newOffset = i; continue; } } //Whether needDiscard() exceeds the maximum number of lookups, the message attribute TRANSACTION_CHECK_TIMES is increased by 1 for each lookup, and the default maximum number of lookups is 15. //needSkip() determines whether the current message exceeds, and the default file expiration time of the system is 72 hours, which can be configured in the broker configuration file. if (needDiscard(msgExt, transactionCheckMax) || needSkip(msgExt)) { listener.resolveDiscardMsg(msgExt); newOffset = i + 1; i++; continue; } if (msgExt.getStoreTimestamp() >= startTime) { log.debug("Fresh stored. the miss offset={}, check it later, store={}", i, new Date(msgExt.getStoreTimestamp())); break; } //Time when messages have been stored long valueOfCurrentMinusBorn = System.currentTimeMillis() - msgExt.getBornTimestamp(); //The time to check the status of the transaction is the time to start the review. It takes some time to open the review after the transaction is submitted. The default is 6 seconds. long checkImmunityTime = transactionTimeout; //Get user-defined review time String checkImmunityTimeStr = msgExt.getUserProperty(MessageConst.PROPERTY_CHECK_IMMUNITY_TIME_IN_SECONDS); if (null != checkImmunityTimeStr) { checkImmunityTime = getImmunityTime(checkImmunityTimeStr, transactionTimeout); //Storage time of transaction messages is less than the interval between open lookups if (valueOfCurrentMinusBorn < checkImmunityTime) { if (checkPrepareQueueOffset(removeMap, doneOpOffset, msgExt)) { newOffset = i + 1; i++; continue; } } } else { if ((0 <= valueOfCurrentMinusBorn) && (valueOfCurrentMinusBorn < checkImmunityTime)) { log.debug("New arrived, the miss offset={}, check it later checkImmunity={}, born={}", i, checkImmunityTime, new Date(msgExt.getBornTimestamp())); break; } } List<MessageExt> opMsg = pullResult.getMsgFoundList(); //Judging whether the conditions for review are satisfied boolean isNeedCheck = (opMsg == null && valueOfCurrentMinusBorn > checkImmunityTime) || (opMsg != null && (opMsg.get(opMsg.size() - 1).getBornTimestamp() - startTime > transactionTimeout)) || (valueOfCurrentMinusBorn <= -1); if (isNeedCheck) { //Need to store a new transaction message from scratch if (!putBackHalfMsgQueue(msgExt, i)) { continue; } //msgExt is the latest offset to send back check messages asynchronously listener.resolveHalfMsg(msgExt); } else { //Pull more completed transaction messages pullResult = fillOpRemoveMap(removeMap, opQueue, pullResult.getNextBeginOffset(), halfOffset, doneOpOffset); log.info("The miss offset:{} in messageQueue:{} need to get more opMsg, result is:{}", i, messageQueue, pullResult); continue; } } //Next in the loop newOffset = i + 1; i++; } //Consumption schedule for updating transaction messages if (newOffset != halfOffset) { transactionalMessageBridge.updateConsumeOffset(messageQueue, newOffset); } //Calculate the latest OP queue consumption progress, update progress long newOpOffset = calculateOpOffset(doneOpOffset, opOffset); if (newOpOffset != opOffset) { transactionalMessageBridge.updateConsumeOffset(opQueue, newOpOffset); } } } catch (Exception e) { e.printStackTrace(); log.error("Check error", e); } }
Annotations are added to each key point. Let's analyze fillOpRemoveMap()
private PullResult fillOpRemoveMap(HashMap<Long, Long> removeMap, MessageQueue opQueue, long pullOffsetOfOp, long miniOffset, List<Long> doneOpOffset) { //Pick up 32 messages PullResult pullResult = pullOpMsg(opQueue, pullOffsetOfOp, 32); if (null == pullResult) { return null; } if (pullResult.getPullStatus() == PullStatus.OFFSET_ILLEGAL || pullResult.getPullStatus() == PullStatus.NO_MATCHED_MSG) { log.warn("The miss op offset={} in queue={} is illegal, pullResult={}", pullOffsetOfOp, opQueue, pullResult); transactionalMessageBridge.updateConsumeOffset(opQueue, pullResult.getNextBeginOffset()); return pullResult; } else if (pullResult.getPullStatus() == PullStatus.NO_NEW_MSG) { log.warn("The miss op offset={} in queue={} is NO_NEW_MSG, pullResult={}", pullOffsetOfOp, opQueue, pullResult); return pullResult; } List<MessageExt> opMsg = pullResult.getMsgFoundList(); if (opMsg == null) { log.warn("The miss op offset={} in queue={} is empty, pullResult={}", pullOffsetOfOp, opQueue, pullResult); return pullResult; } for (MessageExt opMessageExt : opMsg) { //The content stored in the op queue is offset of messages whose half queue transaction messages have commit and rollback Long queueOffset = getLong(new String(opMessageExt.getBody(), TransactionalMessageUtil.charset)); log.info("Topic: {} tags: {}, OpOffset: {}, HalfOffset: {}", opMessageExt.getTopic(), opMessageExt.getTags(), opMessageExt.getQueueOffset(), queueOffset); if (TransactionalMessageUtil.REMOVETAG.equals(opMessageExt.getTags())) { if (queueOffset < miniOffset) { doneOpOffset.add(opMessageExt.getQueueOffset()); } else { removeMap.put(queueOffset, opMessageExt.getQueueOffset()); } } else { log.error("Found a illegal tag in opMessageExt= {} ", opMessageExt); } } log.debug("Remove map: {}", removeMap); log.debug("Done op list: {}", doneOpOffset); return pullResult; }
We plot and analyze the relationship between its half queue and op queue, and how to query messages that need to be reviewed.
Graphic analysis:
removeMap is a message offset of a Map set whose key pair is the half queue and value is the message offset of an op queue. In the graph, there are two pairs (100005, 80002), (100004, 80003).
doneOpOffset is a List collection that stores the message offset of the op queue, with only 8004 in the figure.
When check() loops for messages in the half queue, 100004 is already in the removeMap, skipping the following business and continuing to lookup next 100005 for the next logic to determine whether it has the condition isNeedCheck to lookup messages.
5. Summary
We find that RocketMQ's transactional messages are sent without consumption. The design concept of RocketMQ is that consumers will consume at least once, more than once, because of network delay and other reasons, which will lead to re-sending. We only need to ensure that the messages are put into MQ, and the messages will be consumed and the producers will be guaranteed. Consistency of transaction messages.