1, Filter messages

1. Tag filtration

In most cases, TAG is a simple and useful design to choose the message you want. For example:

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_EXAMPLE");
consumer.subscribe("TOPIC", "TAGA || TAGB || TAGC");

Consumers will receive messages containing TAGA or TAGB or TAGC. But the limitation is that a message can only have one label, which may not work for complex scenarios. In this case, you can filter messages using SQL expressions. The SQL feature can be evaluated by the properties at the time of sending the message. Under the syntax defined by RocketMQ, some simple logic can be implemented. Here is an example:

------------
| message  |
|----------|  a > 5 AND b = 'abc'
| a = 10   |  --------------------> Gotten
| b = 'abc'|
| c = true |
------------
------------
| message  |
|----------|   a > 5 AND b = 'abc'
| a = 1    |  --------------------> Missed
| b = 'abc'|
| c = true |
------------

2. Basic SQL syntax

RocketMQ only defines some basic syntax to support this feature. You can also easily expand it.

• Numerical comparison, such as: >;
• Character comparison, such as: =, < > and IN;
• IS NULL or IS NOT NULL;
• Logical symbols AND, OR, NOT;

The constant support types are:

• Values, such as: 123, 3.1415;
• Characters, such as' abc ', must be enclosed in single quotes;
• NULL, special constant
• Boolean, TRUE or FALSE

Only consumers using push mode can use SQL92 standard sql statements. The interface is as follows:

public void subscribe(finalString topic, final MessageSelector messageSelector)

3. Implement message producer

When sending a message, you can set the properties of the message through putUserProperty

DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
Message msg = new Message("TopicTest",
   tag,
   ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)
);
// Set some properties
msg.putUserProperty("a", String.valueOf(i));
SendResult sendResult = producer.send(msg);

producer.shutdown();

4. Realize message consumers

Use MessageSelector.bySql to filter messages with sql

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");
// Only subscribed messages have this property a, a > = 0 and a < = 3
consumer.subscribe("TopicTest", MessageSelector.bySql("a between 0 and 3");
consumer.registerMessageListener(new MessageListenerConcurrently() {
   @Override
   public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
       return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
   }
});
consumer.start();

2, Transaction message

Process analysis

The figure above shows the general scheme of transaction message, which is divided into two processes: sending and submitting of normal transaction message and compensation process of transaction message.

1. Transaction message sending and submission

(1) Send a message (half message).

(2) The server writes the result in response to the message.

(3) Execute the local transaction according to the sending result (if the write fails, the half message will not be visible to the business at this time, and the local logic will not execute).

(4) Execute Commit or Rollback according to the local transaction status (the Commit operation generates message index, and the message is visible to the consumer)

2. Transaction compensation

(1) For transaction messages without Commit/Rollback (messages with pending status), a "back check" is initiated from the server

(2) Producer receives the callback message and checks the status of the local transaction corresponding to the callback message

(3) Recommit or Rollback according to the local transaction status

Among them, the compensation phase is used to solve the situation of message Commit or Rollback timeout or failure.

3. Transaction message status

The transaction message has three states: commit state, rollback state and intermediate state

• TransactionStatus.CommitTransaction: commits a transaction that allows consumers to consume this message.
• TransactionStatus.RollbackTransaction: rollback transaction, which means that the message will be deleted and not allowed to be consumed.
• TransactionStatus.Unknown: intermediate status, which represents the need to check message queues to determine status.
  ###4.6.1 send transaction message

4. Create transactional producers

Using the TransactionMQProducer class to create a producer and specify a unique producer group, you can set up a custom thread pool to handle these check requests. After executing the local transaction, you need to reply to the message queue according to the execution result. Please refer to the previous section for the returned transaction status.

public class Producer {

    public static void main(String[] args) throws Exception {
        //1. Create a message producer and make a producer group name
        TransactionMQProducer producer = new TransactionMQProducer("group5");
        //2. Specify the Nameserver address
        producer.setNamesrvAddr("39.96.86.4:9876;59.110.158.93:9876");

        //Add transaction listener
        producer.setTransactionListener(new TransactionListener() {
            /**
             * Execute local transactions in this method
             * @param msg
             * @param arg
             * @return
             */
            @Override
            public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
                if (StringUtils.equals("TAGA", msg.getTags())) {
                    return LocalTransactionState.COMMIT_MESSAGE;
                } else if (StringUtils.equals("TAGB", msg.getTags())) {
                    return LocalTransactionState.ROLLBACK_MESSAGE;
                } else if (StringUtils.equals("TAGC", msg.getTags())) {
                    return LocalTransactionState.UNKNOW;
                }
                return LocalTransactionState.UNKNOW;
            }

            /**
             * In this method, MQ performs message transaction status check
             * @param msg
             * @return
             */
            @Override
            public LocalTransactionState checkLocalTransaction(MessageExt msg) {
                System.out.println("News Tag:" + msg.getTags());
                return LocalTransactionState.COMMIT_MESSAGE;
            }
        });

        //3. Start producer
        producer.start();

        String[] tags = {"TAGA", "TAGB", "TAGC"};

        for (int i = 0; i < 3; i++) {
            //4. Create message object, specify Topic, Tag and message body
            /**
             * Parameter 1: Message Topic
             * Parameter 2: message Tag
             * Parameter 3: message content
             */
            Message msg = new Message("TransactionTopic", tags[i], ("Hello World" + i).getBytes());
            //5. Send message
            SendResult result = producer.sendMessageInTransaction(msg, null);
            //Sending State
            SendStatus status = result.getSendStatus();

            System.out.println("Sending result:" + result);

            //Thread sleep for 1 second
            TimeUnit.SECONDS.sleep(2);
        }

        //6. Close producer
        //producer.shutdown();
    }
}

5. Implement the listening interface of transactions

When the semi message is sent successfully, we use the executeloctransaction method to execute the local transaction. It returns one of the three transaction states mentioned in the previous section. The checklocaltransaction method is used to check the status of a local transaction and respond to a check request from a message queue. It is also one of the three transaction states mentioned in the previous section.

6. Restrictions on use

1. Delayed and bulk messages are not supported for transaction messages.
2. In order to avoid the semi queue message accumulation caused by too many single messages being checked, we limit the number of single message checks to 15 by default, but users can modify this limit through the transactionCheckMax parameter in the Broker configuration file. If a message has been checked more than N times (N = transactionCheckMax), Broker will discard the message and print the error log at the same time by default. The user can modify this behavior by overriding the AbstractTransactionCheckListener class.
3. Transaction messages are checked after a specific length of time, such as the transactionMsgTimeout parameter in the Broker configuration file. When sending transaction messages, the user can also change this limit by setting the user property check ﹣ importance ﹣ time ﹣ in ﹣ seconds, which takes precedence over the transactionMsgTimeout parameter.
4. Transactional messages may be checked or consumed more than once.
5. The target subject message submitted to the user may fail, which currently depends on the logging. Its high availability is guaranteed by RocketMQ's own high availability mechanism. If you want to ensure that transaction messages are not lost and transaction integrity is guaranteed, it is recommended to use the synchronous dual write mechanism.
6. The producer ID of a transaction message cannot be shared with the producer ID of another type of message. Unlike other types of messages, transaction messages allow reverse queries, and MQ servers can query consumers through their producer IDs.