According to the description above, the entry to send a transaction message is:

TransactionMQProducer#sendMessageInTransaction: 
public TransactionSendResult sendMessageInTransaction(final Message msg, final Object arg) throws MQClientException {
        if (null == this.transactionListener) {    // @1
            throw new MQClientException("TransactionListener is null", null);
        }

        return this.defaultMQProducerImpl.sendMessageInTransaction(msg, transactionListener, arg);  // @2
    }

Code @1: If transactionListener is empty, an exception is thrown directly.
Code @2: Call the sendMessageInTransaction method of defaultMQProducerImpl.

Next, we will focus on sharing the sendMessageInTransaction method.

DefaultMQProducerImpl#sendMessageInTransaction
public TransactionSendResult sendMessageInTransaction(final Message msg,
           final TransactionListener tranExecuter, final Object arg)  throws MQClientException {

Step1: First of all, I will elaborate on the meaning of parameters.

• final Message msg: message
• TransactionListener tranExecuter: Transaction listener
• Object arg: Other additional parameters

DefaultMQProducerImpl#sendMessageInTransaction
SendResult sendResult = null;
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_TRANSACTION_PREPARED, "true");
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_PRODUCER_GROUP, this.defaultMQProducer.getProducerGroup());
try {
       sendResult = this.send(msg);
} catch (Exception e) {
       throw new MQClientException("send message Exception", e);
}

Step2: Add two attributes to the message attribute: TRAN_MSG, whose value is true, representing the transactional message; PGROUP: the group of senders to which the message belongs, and then send the message synchronously. Before sending a message, the attribute TRAN_MSG of the message is checked. If it exists and the value is true, the message is set to MessageSysFlag.TRANSACTION_PREPARED_TYPE by setting the message system tag.

DefaultMQProducerImpl#sendKernelImpl
final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
       sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
}

SendMessageProcessor#sendMessage
String traFlag = oriProps.get(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (traFlag != null && Boolean.parseBoolean(traFlag)) {
        if (this.brokerController.getBrokerConfig().isRejectTransactionMessage()) {
             response.setCode(ResponseCode.NO_PERMISSION);
             response.setRemark(
                    "the broker[" + this.brokerController.getBrokerConfig().getBrokerIP1()
                        + "] sending transaction message is forbidden");
             return response;
       }
      putMessageResult = this.brokerController.getTransactionalMessageService().prepareMessage(msgInner);
} else {
      putMessageResult = this.brokerController.getMessageStore().putMessage(msgInner);
}

Step3: The Broker side determines the type of message when it receives a message request from the client. If it is a transactional message, call the Transactional Message Service prepareMessage method, otherwise follow the original logic and call the MessageStore putMessage method to store the message in the Broker server.
This section focuses on the principle of transaction message implementation, so we will focus on the prepareMessage method. If you want to know about RocketMQ message storage, you can focus on the author. Source Code Analysis RocketMQ Series.

org.apache.rocketmq.broker.transaction.queue.TransactionalMessageServiceImpl#prepareMessage
public PutMessageResult prepareMessage(MessageExtBrokerInner messageInner) {
        return transactionalMessageBridge.putHalfMessage(messageInner);
 }

Step 4: Transactional Message Service Impl prepareMessage method will be invoked, and then the Transactional Message Bridge prepareMessage method will be invoked.

TransactionalMessageBridge#parseHalfMessageInner
public PutMessageResult putHalfMessage(MessageExtBrokerInner messageInner) {
        return store.putMessage(parseHalfMessageInner(messageInner));
    }

    private MessageExtBrokerInner parseHalfMessageInner(MessageExtBrokerInner msgInner) {
        MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_TOPIC, msgInner.getTopic());
        MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_QUEUE_ID,
            String.valueOf(msgInner.getQueueId()));
        msgInner.setSysFlag(
            MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), MessageSysFlag.TRANSACTION_NOT_TYPE));
        msgInner.setTopic(TransactionalMessageUtil.buildHalfTopic());
        msgInner.setQueueId(0);
        msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgInner.getProperties()));
        return msgInner;
    }

Step5: Back up the original topic name and the original queue ID of the message, then cancel the transaction message label of the message, and reset the theme of the message to RMQ_SYS_TRANS_HALF_TOPIC, with the queue ID fixed to 0. Then call the MessageStore putMessage method to persist the message, where the Transactional Message Bridge bridge class is the process of encapsulating the transaction message, and finally call the MessageStore to complete the message persistence. After the message is stored, it goes back to DefaultMQ Producer Impl # sendMessageInTransaction. After Step2 above, the message is sent to the message server synchronously.

Note: This is the processing logic of the Prepare state of the transaction message. The message is stored in the message server, but the storage is not the original topic, but RMQ_SYS_TRANS_HALF_TOPIC, so the consumer can not consume shen at this time.
The message sent by the producer. See here, if you are familiar with RocketMQ, there must be a "timed task" to pick up the message under this topic, and then "appropriate" time to restore the theme of the message.

DefaultMQProducerImpl#sendMessageInTransaction
switch (sendResult.getSendStatus()) {
            case SEND_OK: {
                try {
                    if (sendResult.getTransactionId() != null) {
                        msg.putUserProperty("__transactionId__", sendResult.getTransactionId());
                    }
                    String transactionId = msg.getProperty(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX);
                    if (null != transactionId && !"".equals(transactionId)) {
                        msg.setTransactionId(transactionId);
                    }
                    localTransactionState = tranExecuter.executeLocalTransaction(msg, arg);
                    if (null == localTransactionState) {
                        localTransactionState = LocalTransactionState.UNKNOW;
                    }

                    if (localTransactionState != LocalTransactionState.COMMIT_MESSAGE) {
                        log.info("executeLocalTransactionBranch return {}", localTransactionState);
                        log.info(msg.toString());
                    }
                } catch (Throwable e) {
                    log.info("executeLocalTransactionBranch exception", e);
                    log.info(msg.toString());
                    localException = e;
                }
            }
            break;
            case FLUSH_DISK_TIMEOUT:
            case FLUSH_SLAVE_TIMEOUT:
            case SLAVE_NOT_AVAILABLE:
                localTransactionState = LocalTransactionState.ROLLBACK_MESSAGE;
                break;
            default:
                break;
        }

Step6: If the message is sent successfully, the TransactionListener#executeLocalTransaction method is called back to execute the local transaction, and the local transaction state is returned to LocalTransactionState with the following enumeration values:

• COMMIT_MESSAGE,
• ROLLBACK_MESSAGE,
• UNKNOW

Note: TransactionListener#executeLocalTransaction executes the local transaction method and returns to the local transaction status after the sender successfully sends the PREPARED message; if the PREPARED message fails to send, the TransactionListener#executeLocalTransaction will not be invoked, and the local transaction message, set to LocalTransactionState.ROLLBACK_MESSAGE, represents the message. Need to be rolled back.

DefaultMQProducerImpl#sendMessageInTransaction
try {
this.endTransaction(sendResult, localTransactionState, localException);
} catch (Exception e) {
log.warn("local transaction execute " + localTransactionState + ", but end broker transaction failed", e);
}

Step 7: Call the endTransaction method to end the transaction (commit or rollback).

DefaultMQProducerImpl#endTransaction
EndTransactionRequestHeader requestHeader = new EndTransactionRequestHeader();
requestHeader.setTransactionId(transactionId);
requestHeader.setCommitLogOffset(id.getOffset());
switch (localTransactionState) {
    case COMMIT_MESSAGE:
         requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_COMMIT_TYPE);
         break;
    case ROLLBACK_MESSAGE:
         requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_ROLLBACK_TYPE);
         break;
     case UNKNOW:
         requestHeader.setCommitOrRollback(MessageSysFlag.TRANSACTION_NOT_TYPE);
         break;
     default:
         break;
}
requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
requestHeader.setTranStateTableOffset(sendResult.getQueueOffset());
requestHeader.setMsgId(sendResult.getMsgId());

Step 8: Assemble end-of-transaction requests. The main parameters are: transaction ID, commit Or Rollback, consumer group, message queue offset, message ID, from Transaction Check. The request sent from here defaults to false. The request processor on the Broker side is EndTransaction Processor.

Step 9: EndTransaction Processor according to transaction submission type: TRANSACTION_COMMIT_TYPE (commit transaction), TRANSACTION_ROLLBACK_TYPE (rollback transaction), TRANSACTION_NOT_TYPE (ignore the request).

So far, the sending process of RocketMQ transaction message has been sorted out in detail, more accurately, the message sending process of Prepare status. The specific flow chart is shown as follows:

At this point, the process of transactional message sending is preliminarily demonstrated. Generally speaking, the two-stage commit method is used in the transactional message sending of RocketMQ. Firstly, when the message is sent, the message of type Prepread is sent first, and then after the message is successfully stored in the message server, the TransactionListener#executeLocalTransaction is called back to execute the local transaction. The state callback function then terminates the transaction according to the return value of the method:
1. COMMIT_MESSAGE: Submit a transaction.
2. ROLLBACK_MESSAGE: Rollback transactions.
3. UNKNOW: Unknown transaction status. When the Broker receives the EndTransaction command, it will not process the message. The message is still in the Prepared type and stored in the queue with the theme RMQ_SYS_TRANS_HALF_TOPIC. Then the message sending process will end. How can these messages be submitted or rolled back?

In order to avoid the need for clients to send submission and rollback commands again, RocketMQ will take a timing task to extract the message from RMQ_SYS_TRANS_HALF_TOPIC, and then return to the client to determine whether the message needs to be submitted or rolled back to complete the declaration cycle of the transaction message, which will be discussed in the next section.