The rocketmq version is 4.9.2. The version of rocketmq spring boot starter is 2.2.1

1, Code

Producer code

• App class

package com.zgd.springboot.demo.simple;

import java.util.Date;

import com.zgd.springboot.demo.simple.mq.MQProducer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;

@SpringBootApplication
public class Application {


  public static void main(String[] args) {
    ApplicationContext context = SpringApplication.run(Application.class, args);
    MQProducer mqProducer = context.getBean(MQProducer.class);
    mqProducer.sendTransactionMsg("Let me test the transaction message 777" + new Date().toLocaleString());
    // for (int i = 0; i < 1; i++) {
    //   mqProducer.sendMsg("let me test" + new date(). Tolocalestring() + "--" + I);
    // }
  }

}

• Normal message sending class

package com.zgd.springboot.demo.simple.mq;

import java.nio.charset.StandardCharsets;
import java.util.Date;
import java.util.UUID;

import org.apache.commons.lang3.RandomUtils;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Component;

import lombok.extern.slf4j.Slf4j;

/**
 * RocketMqListener
 * @date: 2020/11/26
 * @author weirx
 * @version 3.0
 */
@Component
@Slf4j
public class MQProducer {

  @Autowired
  private RocketMQTemplate rocketMQTemplate;

  /**
   * Send normal messages
   */
  public void sendMsg(String msgBody) {
    log.info("Send normal messages");
    rocketMQTemplate.syncSend(
      "queue_test_topic",
      MessageBuilder
        .withPayload(msgBody.getBytes(StandardCharsets.UTF_8))
        .setHeader(RocketMQHeaders.MESSAGE_ID, System.currentTimeMillis() + "")
        .setHeader(
          RocketMQHeaders.KEYS,
          "key-" + System.currentTimeMillis() + ""
        )
        .build()
    );
  }

  public void sendTransactionMsg(String msgBody) {
    log.info("Send transaction message");
    //Simulate that this is the transaction id
    UUID id = UUID.randomUUID();
    int nextInt = RandomUtils.nextInt(0, 100);
    //It will remain stuck in this method until the end of the backcheck or the method is submitted
    System.out.println("send out"+new Date().toLocaleString());
    rocketMQTemplate.sendMessageInTransaction(
      "queue_test_tc_topic",
      MessageBuilder
        .withPayload(msgBody.getBytes(StandardCharsets.UTF_8))
        .setHeader(RocketMQHeaders.TRANSACTION_ID, id)
        .setHeader(RocketMQHeaders.MESSAGE_ID, System.currentTimeMillis() + "")
        .setHeader(
          RocketMQHeaders.KEYS,
          "key-" + System.currentTimeMillis() + ""
        )
        .build(),
      nextInt
    );
    System.out.println("");
    System.out.println("End sending"+new Date().toLocaleString());

  }
}

• Transaction message sending class

package com.zgd.springboot.demo.simple.mq;

import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.rocketmq.spring.annotation.RocketMQTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionState;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHeaders;

@RocketMQTransactionListener
public class TransactionListenerImpl  implements RocketMQLocalTransactionListener {

  private static Map<String, RocketMQLocalTransactionState> STATE_MAP = new HashMap<>();
  private static Map<String, Integer> BACK_QUERY_MAP = new HashMap<>();

  /**
   *  Execute business logic
   *
   * @param message
   * @param o
   * @return
   */
  @Override
  public RocketMQLocalTransactionState executeLocalTransaction(
    Message message,
    Object o
  ) {
    String transId = (String) message
      .getHeaders()
      .get(RocketMQHeaders.TRANSACTION_ID);
    System.out.println("Listening and processing local transactions" + new Date().toLocaleString());
    System.out.println("");
    try {
      System.out.println(
        "Perform operation,Simulate open transaction.affair id: " + transId + " Received the object: " + o
      );
      Thread.sleep(61000);
      // Set transaction status
      if (o.toString().hashCode() % 2 == 0) {
        throw new RuntimeException("Simulation anomaly");
      }
      System.out.println("The simulation completed the transaction.Submit" + new Date().toLocaleString());
      STATE_MAP.put(transId, RocketMQLocalTransactionState.COMMIT);
      // Return transaction status to producer
      return RocketMQLocalTransactionState.COMMIT;
    } catch (Exception e) {
      e.printStackTrace();
    }
    System.out.println("Error simulating transaction.RollBACK " + new Date().toLocaleString());
    STATE_MAP.put(transId, RocketMQLocalTransactionState.ROLLBACK);
    return RocketMQLocalTransactionState.ROLLBACK;
  }

  /**
   * rocket After the executelocetransaction method is executed, it will check back and forth at intervals, even if it has been abnormally rolled back
   * ，After each execution, it will wait for 1min (transactionCheckMax parameter) to execute the next time. By default, 6s (transactionTimeOut parameter) is the minimum time of transaction inspection, and by default, the maximum number of inspections is 15 (transactionCheckMax parameter)
   * @param message
   * @return
   */
  @Override
  public RocketMQLocalTransactionState checkLocalTransaction(Message message) {
    System.out.println("---------");
    MessageHeaders headers = message.getHeaders();
    Set<Entry<String, Object>> entrySet = headers.entrySet();
    System.out.println("Receive and check back" + new Date().toLocaleString());
    for (Entry<String, Object> entry : entrySet) {
      System.out.println(entry.getKey() + " > " + entry.getValue());
    }

    String transId = (String) message
      .getHeaders()
      .get(RocketMQHeaders.TRANSACTION_ID);
    System.out.println(
      "Check back message -> transId = " + transId + ", state = " + STATE_MAP.get(transId)
    );
    System.out.println("");
    if(BACK_QUERY_MAP.get(transId) == null){
      BACK_QUERY_MAP.put(transId,1);
    }else{
      BACK_QUERY_MAP.put(transId,BACK_QUERY_MAP.get(transId)+1);
    }
    //If you check back more than 3 times, roll back directly

    return STATE_MAP.get(transId) == null ? (BACK_QUERY_MAP.get(transId) < 3 ? RocketMQLocalTransactionState.UNKNOWN : RocketMQLocalTransactionState.ROLLBACK) : STATE_MAP.get(transId);
  }
}

Consumer code

• App class
  Normal startup class, ignored
• Common message listening class

package com.zgd.springboot.demo.simple.mq;

import java.nio.charset.StandardCharsets;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.apache.rocketmq.spring.core.RocketMQPushConsumerLifecycleListener;
import org.springframework.stereotype.Component;

import lombok.extern.slf4j.Slf4j;

/**
 * RocketMqListener
 * @date: 2020/11/26
 * consumer Retry (two types: listening and custom consumer)
 * This is the way of listening
 * @author weirx
 * @version 3.0
 */
@Slf4j
@Component
@RocketMQMessageListener(
  topic = "queue_test_topic",
  selectorExpression = "*",
  consumerGroup = "queue_group_test",
  maxReconsumeTimes=2
  
)
public class CommonMQListener implements RocketMQListener<MessageExt> , RocketMQPushConsumerLifecycleListener {

  @Override
  public void onMessage(MessageExt messageExt) {
    byte[] body = messageExt.getBody();
    String msg = new String(body,StandardCharsets.UTF_8);
    int reconsumeTimes = messageExt.getReconsumeTimes();
    log.info("Message received:{} | Re consumption times: {}", msg, reconsumeTimes);
    //The simulation exception will be retried automatically. If the number of retries exceeds: defaultrocketmqllistenercontainer: consume message failed, and then put it into the dead letter queue of the name with topic and DLQ as the suffix
  
    // int i = 1/0;
  }

  @Override
  public void prepareStart(DefaultMQPushConsumer consumer) {
      // The interval of each pull, in milliseconds
      consumer.setPullInterval(3000);
      // Set the number of messages pulled from the queue to 4 each time. By default, there are 4 queues created under each topic. Writequeuenum = readqueuenum = 4, so 4 * 4 = 16 are pulled each time
      consumer.setPullBatchSize(4);
  }
}

• Transaction message listening class

package com.zgd.springboot.demo.simple.mq;

import java.nio.charset.StandardCharsets;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.apache.rocketmq.spring.core.RocketMQPushConsumerLifecycleListener;
import org.springframework.stereotype.Component;

import lombok.extern.slf4j.Slf4j;

/**
 * RocketMqListener
 * @date: 2020/11/26
 * consumer Retry (two types: listening and custom consumer)
 * This is the way of listening
 * @author weirx
 * @version 3.0
 */
@Slf4j
@Component
@RocketMQMessageListener(
  topic = "queue_test_tc_topic",
  selectorExpression = "*",
  //The same consumer group must subscribe to the same topic
  consumerGroup = "queue_group_tc_test",
  maxReconsumeTimes=2
  
)
public class TransactionMQListener implements RocketMQListener<MessageExt> , RocketMQPushConsumerLifecycleListener {

  @Override
  public void onMessage(MessageExt messageExt) {
    byte[] body = messageExt.getBody();
    String msg = new String(body,StandardCharsets.UTF_8);
    int reconsumeTimes = messageExt.getReconsumeTimes();
    String keys = messageExt.getKeys();
    String transactionId = messageExt.getTransactionId();
    log.info("Transaction message received:{} | key: {} | tranId: {} | Re consumption times: {}", msg, keys,transactionId, reconsumeTimes);
    //The simulation exception will be retried automatically. If the number of retries exceeds: defaultrocketmqllistenercontainer: consume message failed, and then put it into the dead letter queue of the name with topic and DLQ as the suffix
  
    // int i = 1/0;
  }

  @Override
  public void prepareStart(DefaultMQPushConsumer consumer) {
      // The interval of each pull, in milliseconds
      consumer.setPullInterval(1000);
      // Set the number of messages pulled from the queue to 4 each time. By default, there are 4 queues created under each topic. Writequeuenum = readqueuenum = 4, so 4 * 4 = 16 are pulled each time
      consumer.setPullBatchSize(4);
      log.info("Initialize transaction consumer");
  }
}

2, Notes

2.1 producers

2.1.1 transaction messages

Transaction messages are not supported from rocketmq after 3.0.8 to before 4.3.0

2.1.1.1 transaction message query

If sent in transaction message mode
The message is sent to a HALF queue of the broker as a HALF message, and the status is UN_KNOW, which has not been delivered to the target queue at this time

Therefore, the message is invisible to the consumer. The message can only be consumed by the consumer after the producer confirms the submission. If it is rolled back, the message will be destroyed
At the same time, rocketmq checks the producer regularly (every minute), and each check will be re delivered to the HALF queue to avoid the failure of submission or rollback

2.1.1.2 message duplication

At first, to facilitate the test, I set the sleep time of local transactions to 120s, and it was found that there were always two duplicate messages. After careful investigation, it was found that:

When the local transaction takes a long time, it will trigger a callback and insert a duplicate message into the HALF topic

• The first message put into the destination queue
  When the local transaction is completed and the COMMIT is returned, it will be thrown into the target topic for consumer consumption
• The second message put into the target queue
  In the case of a scheduled task, when MQ checks back again and gets a reply from the COMMIT, it will also drop a message to the target queue, resulting in duplicate messages

As shown in the figure below:

Avoidance: try to ensure that local transactions are completed in one minute

2.2 consumers

2.2.1 consumption retry

If a consumer throws an exception during consumption, it will try to consume again, and the interval will increase with the number of times. By default, it will retry 16 times. After failure, it will be thrown into the dead letter queue

2.2.2 batch pulling

The consumer can set the PullBatchSize parameter to indicate the number of messages pulled from each queue at one time. By default, there are 4 brokers (the broker creates 8 queues by default, but the producer's DefaultMQProducer creates 4 queues by default, whichever is less)

2.2.3 the message cannot be consumed. The status of the message is NOT_CONSUME_YET

I was not very clear about this at the beginning. In the following notes, the relationship between the consumer group and topic is directly copied and used when testing transaction consumption, and the topic is changed. That is, two topics are subscribed to the same group

@RocketMQMessageListener(
  topic = "queue_test_topic",
  selectorExpression = "*",
  consumerGroup = "queue_group_test",
  maxReconsumeTimes=2
  
)

As a result, there was a problem that some messages mentioned above could not be consumed. After looking at the console, the consumer found a queue_ test_ There are 0 consumer terminals for topic, but there are no queues_ test_ tc_ Topic has two consumer terminals
Look at the queue again_ test_ In the topic queue, there is naturally no consumer, but the queue_ test_ tc_ In the topic queue, only 2 of the 4 queues (4 by default) have subscriptions

principle
The subscription relationship must be the same for the same group

In RocketMQ, the consumer group and subscription relationship information are a Map. If consumer instances a and b are in the same group and subscribe to a and b respectively, the B-b relationship loaded late at the time of registration will overwrite A-a, resulting in both a and b subscribing to b
The above situation is that ordinary consumers and transaction consumers finally subscribe
queue_test_tc_topic

Error condition:

Normal conditions:

• If the same group and multiple consumers subscribe to the same topic: Yes, the consumers will be automatically assigned to each queue of the topic
• If the same group and multiple consumers subscribe to different topics: No, it will be confused, resulting in the coverage of the topic subscription relationship and the incomplete allocation of the matching relationship of the queue in the topic. As a result, the message cannot be consumed
• If the same two groups have one consumer and subscribe to two topic s: Yes, the only consumer will be automatically matched to all queue queues
• If different groups subscribe to the same topic: Yes. RocketMQ will automatically assign consumers of each group to each queue of the topic and consume messages at the same time