Next: RocketMQ entry to earth (1) novice can also understand the principle and actual combat!

1, Origin of transaction message

1. Case study

Quoting official shopping cases:

At the same time, you need to add 100 points to the downstream account to ensure that 100 yuan will be deducted from the system. Account system and points system are two independent systems, one to reduce 100 yuan, the other to increase 100 points. As shown in the figure below:

 

2. Questions

• The account service has successfully deducted money, and the notification points system has also succeeded. However, when the points are increased, it fails, and the data is inconsistent.

• The account service deducted money successfully, but the notification points system failed, so the points will not increase, and the data is inconsistent.

3. Programme

RocketMQ's solution to the first problem is: if the consumption fails, it will be automatically retried. If the consumption fails after several retries, the situation needs to be solved manually, such as putting it in the dead letter queue and manually checking the cause for processing.

RocketMQ's solution to the second problem is: if you succeed in withholding money but fail to write a message to mq, RocketMQ will roll back the message. At this time, we can also roll back our deduction operation.

2, Principle of transaction message

1. Principle diagram

 

2. Detailed process

1.Producer sends Half Message to broker.

• After the Half Message is sent successfully, the local transaction is started.

• If the local transaction is successful, it returns commit, and if it fails, it returns rollback. (in the callback method of transaction messages, the developer decides to commit or rollback)

Producer sends the previous commit or rollback to the broker. There are two situations:

1. If the broker receives a commit/rollback message:

• If a commit is received, the broker thinks that the whole transaction is OK and the execution is successful. Then a message will be sent to the Consumer for consumption.

• If a rollback is received, the broker thinks that the local transaction has failed. The broker will delete the Half Message and send it to the Consumer.

2. If the broker does not receive the message (if the execution of the local transaction is suddenly down and the local transaction execution result returns unknow n, it will be handled as if the broker did not receive the confirmation message.) :

• The broker will regularly check the execution result of the local transaction: if the result is that the local transaction has been executed, it will return commit; if not, it will return rollback.

• The results of the Producer side back check are sent to the broker. If the broker receives a commit, it is considered that the whole transaction has been successfully executed. If it is a rollback, it is considered that the local transaction has failed. The broker deletes the Half Message and does not send it to the consumer. If the broker does not receive the result of the back query (or the query is unknow n), the broker will perform repeated back checks regularly to ensure that the final transaction result is found. The time interval and times of repeated back check can be matched.

3, Transaction message implementation process

1. Implementation process

In brief, the transaction message is a listener with a callback function. In the callback function, we perform business logic operations, such as giving the account - 100 yuan, and then sending a message to the integral mq. At this time, if the account - 100 succeeds and the message is sent to mq successfully, the message status is set to commit. At this time, the broker will send the semi message to the real topic. At the beginning, it was stored in the semi message queue, not in the real topic queue. It will not be transferred until commit is confirmed.

2. Remedial plan

If the transaction fails to respond immediately due to interruption or other network reasons, RocketMQ is treated as unknown. RocketMQ transaction message also provides a remedy: regularly query the transaction status of transaction message. This is also a callback function, which can be used to make compensation. The developer of the compensation logic can write it by himself. If it succeeds, he will return to commit and it will be finished.

4, Transaction message code instance

1. Code

package com.chentongwei.mq.rocketmq;

import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.client.producer.TransactionListener;
import org.apache.rocketmq.client.producer.TransactionMQProducer;
import org.apache.rocketmq.client.producer.TransactionSendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;

import java.util.Date;

/**
 * Description:
 *
 * @author TongWei.Chen 2020-06-21 11:32:58
 */
public class ProducerTransaction2 {
    public static void main(String[] args) throws Exception {
        TransactionMQProducer producer = new TransactionMQProducer("my-transaction-producer");
        producer.setNamesrvAddr("124.57.180.156:9876");

        // Callback
        producer.setTransactionListener(new TransactionListener() {
            @Override
            public LocalTransactionState executeLocalTransaction(Message message, Object arg) {
                LocalTransactionState state = null;
                //msg-4 return COMMIT_MESSAGE
                if(message.getKeys().equals("msg-1")){
                    state = LocalTransactionState.COMMIT_MESSAGE;
                }
                //msg-5 return ROLLBACK_MESSAGE
                else if(message.getKeys().equals("msg-2")){
                    state = LocalTransactionState.ROLLBACK_MESSAGE;
                }else{
                    //Back here unknown The purpose of this paper is to simulate the sudden outage of local transaction (or the scenario where the local execution succeeds and the confirmation message fails to be sent)
                    state = LocalTransactionState.UNKNOW;
                }
                System.out.println(message.getKeys() + ",state:" + state);
                return state;
            }

            /**
             * Back look up method of transaction message
             */
            @Override
            public LocalTransactionState checkLocalTransaction(MessageExt messageExt) {
                if (null != messageExt.getKeys()) {
                    switch (messageExt.getKeys()) {
                        case "msg-3":
                            System.out.println("msg-3 unknow");
                            return LocalTransactionState.UNKNOW;
                        case "msg-4":
                            System.out.println("msg-4 COMMIT_MESSAGE");
                            return LocalTransactionState.COMMIT_MESSAGE;
                        case "msg-5":
                            //The query found that the local transaction execution failed, and the message needs to be rolled back.
                            System.out.println("msg-5 ROLLBACK_MESSAGE");
                            return LocalTransactionState.ROLLBACK_MESSAGE;
                    }
                }
                return LocalTransactionState.COMMIT_MESSAGE;
            }
        });

        producer.start();

        //Simulate sending 5 messages
        for (int i = 1; i < 6; i++) {
            try {
                Message msg = new Message("transactionTopic", null, "msg-" + i, ("Test, this is transaction message! " + i).getBytes());
                producer.sendMessageInTransaction(msg, null);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

 

2. Results

msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW

msg-3 unknow
msg-3 unknow
msg-5 ROLLBACK_MESSAGE
msg-4 COMMIT_MESSAGE

msg-3 unknow
msg-3 unknow
msg-3 unknow
msg-3 unknow

3. Control desk

4. Result analysis

• Only msg-1 and msg-4 were sent successfully. Msg-4 preceded msg-1 because msg-1 succeeded first, and msg-4 succeeded only after checking back. It's in reverse chronological order.

• First, output five results, corresponding to five messages

msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW

• Then it enters the backcheck, msg-3 is still unknow n, msg-5 rolls back, and msg-4 commits the transaction. So msg-4 can be seen in the control console.

• After a period of time, we checked msg-3 again and found that it was still unknow n, so we kept checking.

5, Questions

Question: can Spring transaction and regular distributed transaction not work? Is Rocketmq's transaction superfluous?

MQ is used to decouple. Previously, distributed transactions directly operated the account system and credit system. However, strong coupling exists. If an MQ is inserted in the middle, the account system will send a message to the MQ after the operation. At this time, as long as the sending is successful, it will be submitted, and if it fails, it will be rolled back. How to ensure this step depends on the transaction. Moreover, RocketMQ is used for distributed transactions.

6, Sequential message interpretation

1. Overview

RocketMQ messages are stored in the Topic queue. The queue itself is a FIFO (First Int First Out) FIFO queue. So a single queue can guarantee the order.

But the problem is that a topic has N queues, and the advantages of this design are also obvious. It naturally supports clustering and load balancing, and distributes massive data evenly to each queue. When you send 10 messages to the same topic, these 10 messages will be automatically distributed among all the queues under the topic. Therefore, it is not necessarily the queue to be consumed first and which queue to consume later This leads to disordered consumption.

2. Illustration

 

3. Re analysis

A Producer sends four messages m1, m2, m3 and m4 to the topic. The topic has four queues. Due to its own load balancing strategy, one message is stored on the four queues. m1 stored on queue1, m2 stored on queue2, m3 stored on queue3, m4 stored on queue4 are consumed by multithreading, so it is impossible to guarantee which queue or message is consumed first. For example, the order of sending is m1, m2, m3 and m4. However, since the internal consumption of Consumer is multi-threaded, it may consume queue 4 first m4 on the column, then m1, which results in disorder.

7, Sequential messaging solution

1. Scheme 1

Very simple, the key to the problem is that there are messages in multiple queues. When I consume, I don't know which queue has the latest messages. Then there is the idea. When sending messages, if you want to ensure the order, send them to a queue for me. When consuming, because only that queue has messages and the queue is FIFO, FIFO, the normal consumption is over.

It's perfect. Moreover, RocketMQ also provides us with this api (message queue selector) for selecting queue when sending messages. Code directly.

2. Code one

2.1 producers

import java.util.List;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MessageQueueSelector;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageQueue;

/**
 * message sender 
 */
public class Producer5 {
    public static void main(String[] args)throws Exception {
        DefaultMQProducer producer = new DefaultMQProducer("my-order-producer");
        producer.setNamesrvAddr("124.57.180.156:9876");     
        producer.start();
        for (int i = 0; i < 5; i++) {
            Message message = new Message("orderTopic", ("hello!" + i).getBytes());
            producer.send(
                    // The message to be sent
                    message,
                    // queue Selector, to topic Which of them queue To write a message
                    new MessageQueueSelector() {
                        // Select one manually queue
                        @Override
                        public MessageQueue select(
                                // current topic All that's in it queue
                                List<MessageQueue> mqs, 
                                // The specific message to be sent
                                Message msg,
                                // Corresponding to send() Inside args，That's the zero in front of 2000
                                // In the actual business, you can change 0 into the primary key of the actual business system, such as the order number, and then do it here hash Make a choice queue Etc. There are many things that can be done. I will use the first one here for demonstration queue，So no arg. 
                                Object arg) {
                            // To the fixed one queue Write a message in it. For example, this is to the first one queue Write a message in it
                            MessageQueue queue = mqs.get(0);
                            // Selected queue
                            return queue;
                        }
                    },
                    // Custom parameter: 0
                    // 2000 Represents the 2000 ms timeout
                    0, 2000);
        }
    }
}

 

2.2 consumers

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.*;
import org.apache.rocketmq.common.message.MessageExt;

import java.util.List;

/**
 * Description:
 *
 * @author TongWei.Chen 2020-06-22 11:17:47
 */
public class ConsumerOrder {
    public static void main(String[] args) throws Exception {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("my-consumer");
        consumer.setNamesrvAddr("124.57.180.156:9876");
        consumer.subscribe("orderTopic", "*");
        consumer.registerMessageListener(new MessageListenerOrderly() {
            @Override
            public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
                for (MessageExt msg : msgs) {
                    System.out.println(new String(msg.getBody()) + " Thread:" + Thread.currentThread().getName() + " queueid:" + msg.getQueueId());
                }
                return ConsumeOrderlyStatus.SUCCESS;
            }
        });

        consumer.start();
        System.out.println("Consumer start...");
    }
}

 

2.3 output results

Consumer start...
hello!0 Thread:ConsumeMessageThread_1 queueid:0
hello!1 Thread:ConsumeMessageThread_1 queueid:0
hello!2 Thread:ConsumeMessageThread_1 queueid:0
hello!3 Thread:ConsumeMessageThread_1 queueid:0
hello!4 Thread:ConsumeMessageThread_1 queueid:0

3. Situation two

For example, your new demand: put all the unpaid orders in queue1, the paid orders in queue2, and the orders with abnormal payment into queue3. Then you should ensure that each queue is in order when you consume. You can't consume queue1 directly to queue2, but you should consume one by one.

At this time, the idea is to use the user-defined parameter arg when sending a message. The message body must contain the payment status. If it is judged that it is not paid, select queue1, and so on. This ensures that each queue contains only messages with the same status. So consumers are currently multi-threaded consumption, certainly out of order. Three queues are randomly consumed. The solution is simpler. Change the number of threads on the consumer side to 1, so that the queue is FIFO and he consumes them one by one. RocketMQ also provides us with such an api, as follows:

// Maximum number of threads 1
consumer.setConsumeThreadMax(1);
// Minimum number of threads
consumer.setConsumeThreadMin(1);