Practical application
maven dependency
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.9.1</version>
</dependency>
General message
message sending
1. The Producer sends a synchronization message
This reliable synchronous sending method is widely used, such as important message notification and short message notification.
public static void syncProducer() throws Exception{
// Instantiate message Producer
DefaultMQProducer producer = new DefaultMQProducer("Group_A");
// Set the address of the NameServer
producer.setNamesrvAddr("localhost:9876");
//Start Producer instance
producer.start();
for (int i = 0; i < 100; i++) {
// Create a message and specify Topic, Tag and message body
Message message = new Message("TopicTest", "TagA",
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
// Send messages to a Broker
SendResult sendResult = producer.send(message);
System.out.println(sendResult);
}
// Stop sending messages and close the Producer instance
producer.shutdown();
}
2. Send asynchronous message
Asynchronous messages are usually used in business scenarios that are sensitive to response time, that is, the sender cannot tolerate waiting for a Broker's response for a long time.
public static void asyncProducer() throws Exception{
DefaultMQProducer producer = new DefaultMQProducer("Group_A");
producer.setNamesrvAddr("localhost:9876");
producer.start();
// retry count
producer.setRetryTimesWhenSendAsyncFailed(0);
// Instantiate the countdown calculator based on the number of messages
final CountDownLatch2 countDownLatch2 = new CountDownLatch2(messageCount);
for(int i = 0; i < messageCount; i++){
final int index = i;
Message message = new Message("TopicTest", "TagB",
"OrderID909", "Hello World".getBytes(RemotingHelper.DEFAULT_CHARSET));
// SendCallBack accepts a callback that returns results asynchronously
producer.send(message, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
System.out.printf("%-10d OK %s %n", index, sendResult.getMsgId());
}
@Override
public void onException(Throwable throwable) {
System.out.printf("%-10d Exception %s %n", index, throwable);
throwable.printStackTrace();
}
});
}
// Wait for 5s
countDownLatch2.await(5, TimeUnit.SECONDS);
producer.shutdown();
}
3. Send one-way message
This method is mainly used in scenarios that do not particularly care about sending results, such as log sending.
public static void onewayProducer() throws Exception{
DefaultMQProducer producer = new DefaultMQProducer("Group_A");
producer.setNamesrvAddr("localhost:9876");
producer.start();
for (int i = 0; i < messageCount; i++) {
Message message = new Message("TopicTest", "TagC",
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
// Send a one-way message without returning results
producer.sendOneway(message);
}
producer.shutdown();
}
Consumption news
public static void consumer() throws MQClientException {
// Instantiate consumer
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("Consumer_GroupA");
//Set NameServer address
consumer.setNamesrvAddr("localhost:9876");
// Subscribe to one or more topics and tags to filter messages to be consumed
consumer.subscribe("TopicTest", "*");
// Register the callback implementation class to handle the messages pulled back from the broker
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgList, ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgList);
// Mark that the message has been successfully consumed
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//Start consumer instance
consumer.start();
System.out.println("Consumer Started");
}
Send delay message
You only need to set the DelayTimeLevel delay level for the message
for (int i = 0; i < totalMessagesToSend; i++) {
Message message = new Message("TestTopic", ("Hello scheduled message " + i).getBytes());
// Set the delay Level 3, and the message will be sent after 10s (now only a few fixed times are supported, see delaytimelevel for details)
message.setDelayTimeLevel(3);
// send message
producer.send(message);
}
Sequential message
Message ordering means that messages can be consumed (FIFO) according to the sending order of messages. RocketMQ can strictly guarantee message order, which can be divided into partition order or global order.
The principle of sequential consumption is analyzed. By default, Round Robin polling will be used to send messages to different queues (partition queues); When consuming messages, pull messages from multiple queues. In this case, the order of sending and consumption cannot be guaranteed. However, if the control sending sequence messages are only sent to the same queue in turn, and the consumption is only pulled from this queue in turn, the sequence is guaranteed. When there is only one queue for sending and consuming, it is globally ordered; If multiple queues participate, the partition is ordered, that is, the messages are ordered relative to each queue.
In local ordering: a MessageQueue can only be consumed by one consumer and can only be consumed by a single thread. However, this consumer can start multithreading and consume multiple messagequeues at the same time.
Sequential message production
The following is an example of order partitioning. The sequential process of an order is: create, pay, push and complete.
When producing messages, messages with the same order number will be sent to the same MessageQueue successively; During consumption, the same queue must be obtained by the same OrderId.
/**
* Sequential message production
*/
public static void orderProducer() throws Exception{
DefaultMQProducer producer = new DefaultMQProducer("ProducerA");
producer.setNamesrvAddr("localhost:9876");
producer.start();
String[] tags = new String[]{"TagA", "TagC", "TagD"};
// Order list
List<OrderStep> orderList = new Producer().buildOrders();
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String dateStr = sdf.format(date);
for (int i = 0; i < 10; i++) {
// Add a time prefix
String body = dateStr + " Hello RocketMQ " + orderList.get(i);
Message msg = new Message("TopicTest", tags[i % tags.length], "KEY" + i, body.getBytes());
SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Long id = (Long) arg; //Select the send queue according to the order id
long index = id % mqs.size(); //Take mold
return mqs.get((int) index);
}
}, orderList.get(i).getOrderId());//Order id
System.out.println(String.format("SendResult status:%s, queueId:%d, body:%s",
sendResult.getSendStatus(),
sendResult.getMessageQueue().getQueueId(),
body));
}
producer.shutdown();
}
Sequential message consumption
/**
* Sequential message consumption
*/
public static void orderConsumer() throws Exception{
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ConsumerA");
consumer.setNamesrvAddr("localhost:9876");
/**
* Set whether the Consumer starts consumption at the head of the queue or at the end of the queue for the first time < br >
* If it is not started for the first time, continue to consume according to the last consumption position
*/
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.subscribe("TopicTest", "TagA || TagC || TagD");
consumer.registerMessageListener(new MessageListenerOrderly() {
Random random = new Random();
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
for (MessageExt msg : msgs) {
// You can see that each queue has a unique consumer thread to consume, and the orders are ordered for each queue (partition)
System.out.println("consumeThread=" + Thread.currentThread().getName() + "queueId=" + msg.getQueueId() + ", content:" + new String(msg.getBody()));
}
try {
//Simulating business logic processing
TimeUnit.SECONDS.sleep(random.nextInt(10));
} catch (Exception e) {
e.printStackTrace();
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
System.out.println("Consumer Started.");
}
- Key points: each queue has a unique consumer thread to consume, and orders are ordered for each queue (partition)
From the consumer.registerMessageListener(new MessageListenerOrderly() code, we can know that sequential messages use MessageListenerOrderly to tell consumers to consume messages sequentially, and only a single thread can consume the same queue. Ordinary messages use messagelistenercurrently to consume messages concurrently.
Transaction message
Transaction messages have three statuses: commit status, rollback status and intermediate status:
- TransactionStatus.CommitTransaction: commit a transaction that allows the consumer to consume this message.
- TransactionStatus.RollbackTransaction: rollback transaction, which means that the message will be deleted and cannot be consumed.
- TransactionStatus.Unknown: intermediate status, which means that the message queue needs to be checked to determine the status.
Create transactional producer
Using the TransactionMQProducer class to create a producer and specify a unique producer group, you can set a custom thread pool to process these check requests. After executing a local transaction, you need to reply to the message queue according to the execution results.
public static void transactionProducer() throws Exception{
TransactionListener transactionListener = new TransactionListenerImpl();
TransactionMQProducer producer = new TransactionMQProducer("TransactionProducer");
ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory() {
@Override
public Thread newThread(Runnable runnable) {
Thread thread = new Thread(runnable);
thread.setName("client-transaction-msg-check-thread");
return thread;
}
});
producer.setExecutorService(executorService);
producer.setTransactionListener(transactionListener);
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
for (int i = 0; i < 10; i++) {
try {
Message message = new Message("MyTopic", tags[i % tags.length], "KEY" + i,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
TransactionSendResult sendResult = producer.sendMessageInTransaction(message, null);
System.out.println(sendResult);
Thread.sleep(10);
} catch (MQClientException | UnsupportedEncodingException e) {
e.printStackTrace();
}
}
for (int i = 0; i < 100000; i++) {
Thread.sleep(1000);
}
producer.shutdown();
}
Implement the transaction listening interface
When sending a semi successful message, we use the executelocetransaction method to execute the local transaction. It returns one of the three transaction states mentioned in the previous section. The checkLocalTransaction method is used to check the local transaction state and respond to the check request of the message queue. It also returns one of the three transaction states mentioned in the previous section.
public class TransactionListenerImpl implements TransactionListener {
private AtomicInteger transactionIndex = new AtomicInteger(0);
private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>();
/**
* Execute local transactions
*/
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
int value = transactionIndex.getAndIncrement();
int status = value % 3;
localTrans.put(msg.getTransactionId(), status);
//Return unknown and ask the Broker to check the transaction status
return LocalTransactionState.UNKNOW;
}
/**
* Check local transaction status
*/
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
Integer status = localTrans.get(msg.getTransactionId());
if(null != status){
switch (status){
case 0:
return LocalTransactionState.UNKNOW;
case 1:
return LocalTransactionState.COMMIT_MESSAGE;
case 2:
return LocalTransactionState.ROLLBACK_MESSAGE;
}
}
return LocalTransactionState.COMMIT_MESSAGE;
}
}
Transaction message flow
It is divided into two processes: sending and submitting normal transaction messages and compensating transaction messages.
1. Transaction message sending and submission:
(1) Send a message (half message).
(2) The server response message writes the result.
(3) Execute the local transaction according to the sending result (if the write fails, the half message is not visible to the business, and the local logic will not execute).
(4) Execute Commit or Rollback according to the local transaction status (the Commit operation generates a message index, and the message is visible to the consumer)
2. Compensation process (back check):
(1) For transaction messages without Commit/Rollback (messages in pending status), initiate a "backcheck" from the server
(2) The producer receives the callback message and checks the status of the local transaction corresponding to the callback message
(3) Re Commit or Rollback according to the local transaction status
The compensation phase is used to solve the timeout or failure of the message Commit or Rollback.
Best practices
Use of Tags
As far as possible, an application uses a Topic, and the message subtype can be identified by tags. Tags can be freely set by the application. Only when the producer sets tags when sending messages, can the consumer use tags to filter messages through the broker when subscribing to messages: message.setTags("TagA").
Use of Keys
The unique identification code of each message at the business level should be set to the keys field to locate the message loss problem in the future. The server will create an index (hash index) for each message , the application can query the content of this message and who consumes it through topic and key. Since it is a hash index, please ensure that the key is as unique as possible, so as to avoid potential hash conflicts.
Transaction message
When executing the local transaction executelocal transaction, if the business execution fails, you can explicitly notify the Rollback and directly return to the Rollback; If the business is successful, it is not recommended to return Commit directly, but to return unknown.
Then, when the transaction is checked back to checkLocalTransaction, if the transaction is successful, the Commit is returned. If it is not clear that the local transaction is successful, unknown is returned, and the server checks back 15 times by default.
Message sending failure processing method
The send method of Producer supports internal retry. The retry logic is as follows:
- Retry up to 2 times.
- If the synchronous mode transmission fails, it will be rotated to the next Broker. If the asynchronous mode transmission fails, it will only be retried in the current Broker. The total time-consuming of this method does not exceed the value set by sendMsgTimeout, which is 10s by default.
- If the message itself sent to the broker generates a timeout exception, it will not be retried.
The above strategy also ensures that the message can be sent successfully to a certain extent. If the business requires high message reliability, it is recommended to add corresponding retry logic: for example, when calling the send synchronization method fails to send, try to store the message in db, and then the background thread will retry regularly to ensure that the message will reach the Broker.