Preface

Before introducing a technology, it must be clear what problems the technology can solve for the project. Before I understood message queue, I thought that message queue was mainly used to send messages such as SMS and email (asynchronous decoupling), but I found my understanding was wrong only after in-depth understanding. MQ's function is not only reflected in the specific messages received by some users, but also can be used for data transmission of other applications, general business processing, etc.
Message queue literally means to store messages in the queue and consume messages according to the characteristics of queue FIFO (first in, first out). In actual development, it is a cross process communication mechanism, which is used for message passing between applications.

Advantages and disadvantages and application scenarios to be understood before MQ is introduced

The main advantages of MQ are decoupling, asynchrony and peak clipping. Here is a simple scenario to reflect these characteristics.

In the microservice project, the system will be vertically split according to the core business and then deployed separately. In the figure above, the main responsibilities of each system in the order business are as follows:

• Order system: create an order and send the order message (such as order id, user data) to MQ
• MQ: limit the number of order requests processed per second (for example, if 2000 requests are received per second, but the database can only process 1000, only 1000 will be processed, and those that cannot be processed will be piled up in the message queue first)
• Logistics system: create order logistics information
• Integration system: update of user's shopping integration information

Imagine the problems in creating an order process when MQ is not available in the above scenario:

• After the order system has created the order information, it needs to call the business interface on the logistics system and the integration system. The system is seriously coupled (decoupled)
• If the order system does not call the interfaces of other systems through threads, it will waste a lot of time to wait for the return synchronously (asynchronous, to avoid the trouble of creating thread calls)
• Too many requests from the database during the peak period of users can not be processed, which leads to application crash (peak shaving)

There are two sides to everything. Although MQ can solve many problems for the system, it will also introduce some problems, such as:

• With the increase of system complexity, it is necessary to consider the problems of repeated message consumption and message loss
• Data consistency problems, such as how to roll back compensation in case of exceptions in writing to the warehouse of logistics or inventory system in the above example

After understanding some features of MQ, we will discuss the following scenarios that are suitable for using MQ:

• The upstream system doesn't care about the downstream execution results (for example, the user system sends an email to the user through MQ after the user registration is successful, but the user system doesn't care if the email is unsuccessful)
• Scheduled tasks that depend on data (if the payment is not made within 24 hours after the following order, the order will be cancelled; if the merchant does not process the refund within 72 hours after the application is made, the refund will be automatically made)

Solutions to some problems after introducing MQ

• Message repeat consumption (to guarantee message idempotence)

  Idempotency: no matter how many times a request for the same operation has the same result, the specific embodiment in MQ is that the same message will be consumed once no matter how many times it is sent.

  Because of the network jitter (delay), the problem of repeated message sending is inevitable. If the idempotence of the message is not guaranteed in the consumption, repeated consumption may occur, resulting in multiple consumption of the same message and multiple writing to the library. A common practice is to add a unique ID to the message when it is consumed
  Query whether the message record exists in the database according to the ID. if there is no further insert message, no insert consumption will occur.
  When the time interval between generation and consumption is not long, Redis can be used to improve the efficiency of message idempotence, such as:

  1. Before consumption, the consumer checks whether the message exists in redis according to the ID
  2. If the message does not exist, it will be consumed and written to redis. If the message exists, it will not be consumed and returned
     About message ID:
  3. Every message of RocketMQ will be provided with a globally unique ID
  4. If message middleware does not generate ID (such as kafka), consider some ID services (such as snowflake algorithm) to generate globally unique ID
  5. Recommendation ID is not associated with actual business

For example, at present, the message center application in personal work is based on the MongoDB+RocketMQ technical architecture. MongoDB is responsible for storing messages (mainly Sms, Email, etc.) sent by each application. Before each consumption, query Mongo through RocketMQ's Message ID to ensure message idempotency and avoid repeated consumption. After successful consumption, update the message status in DB.

• Message loss (message reliability)

  The message loss meanings of MQ components are different, resulting in different solutions. Take the message delivery model (producer > broker > consumer) of kafka and rocket as an example:
  • Producer: the message is not persisted to the Broker, or the consumer has not successfully consumed the message. Kafka can be solved by changing the ack configuration, and the message sending status code will be returned in rocketMQ.
  • Broker: the message has been successfully sent to me, but I lost it for some reasons (different MQ may have different reasons due to mechanism problems). If it is for hardware reasons (such as downtime, disk damage), I suggest you copy several me (cluster deployment)
  • Consumer: I got the news, but I failed or hung up in the middle and didn't tell Broker. It can be solved by the ACK mechanism of each MQ middleware.

Simple example technology framework and business model based on RocketMQ

The following is a simple message center project case based on MongoDB+RocketMQ+Eureka+Spring Cloud Config and combined with the problems in MQ. The main functions of each component in the project are as follows:

• Spring Cloud Config: Message configuration (such as topic, ConsumerGroup, producer group) center.
• Eureka: application service registry, which is responsible for the discovery and delivery of services in the project.
• Mongodb: because the transaction relationship of messages is not strong and the documents in mongodb format are free (json storage, add or delete fields at will), mongodb is used to store messages sent by various applications (mainly Sms, Email, etc.), query Mongo through RocketMQ's Message ID before each consumption to ensure message idempotence and avoid repeated consumption, and save the messages after successful consumption.
• RocketMQ: message receiving, storing and sending.

The following figure shows the application relationship model of the project:

Message center application: business processing application of unified general message, such as message sending, email sending, employee service number pushing, etc
Questionnaire application: responsible for the distribution of employee questionnaire. In this case, it is just a simple message sending test application
common: store general application classes, such as SMS message and message constant
Config server properties: configuration storage directory of configuration center
Because this project is mainly used to demonstrate some MQ functions and solutions to problems in use, the coding part is relatively simple.

Application encoding

• General module code (common)

  The general module mainly stores the general application classes (such as entity, constant, configuration, function, etc.).
  MessageConstant: maintain message constant

  public interface MessageConstant {
  
      interface System {
          String QUESTION = "QUESTION";
      }
  
      interface Topic {
          String SMS_TOPIC = "rocketmq.topic.sms";
          String SMS_TOPIC_TEMPLATE = "${rocketmq.topic.sms}";
          String MAIL_TOPIC = "rocketmq.topic.mail";
          String MAIL_TOPIC_TEMPLATE = "${rocketmq.topic.mail}";
      }
  
      interface Producer {
          String SMS_GROUP_TEMPLATE = "${rocketmq.producer.group.sms}";
          String MAIL_GROUP_TEMPLATE = "${rocketmq.producer.group.mail}";
      }
  
      interface Consumer {
          String SMS_GROUP_TEMPLATE = "${rocketmq.consumer.group.sms}";
          String MAIL_GROUP_TEMPLATE = "${rocketmq.consumer.group.mail}";
      }
  }
  

  BaseMessage: basic message class. All the general messages used need to be integrated to facilitate the management of unified information

  @Data
  @Accessors(chain = true)
  public abstract class BaseMessage implements Serializable {
  
      /**
       * Message source system: {@ link io.wilson.common.message.constant.MessageConstant.System}
       */
      private String system;
  }
  
  

  SmsMessage: General SMS message class, SMS content data carrier

  @EqualsAndHashCode(callSuper = true)
  @Data
  @Accessors(chain = true)
  @ToString(callSuper = true)
  public class SmsMessage extends BaseMessage {
  
      /**
       * SMS creation user
       */
      private String createUserId;
      /**
       * Receiving SMS users
       */
      private String toUserId;
      /**
       * Phone number
       */
      private String mobile;
      /**
       * Content of short message
       */
      private String content;
  
  }
  

• Message center application (question APP)

  Before message center encodes messages, it needs to confirm how message center should process messages. The business environment of the project is that each application may need to send some SMS messages, e-mails, service number messages, etc. the business processing of the same message is consistent, so the main process of message reception and consumption in the message center is as follows:

  • Ensure message idempotence (query database uses existing message records to avoid repeated consumption)
  • Message business processing
  • Message log entry

  In this project, different message types are stored in different mongodb collections (the same as Mysql table concept), but a message log class MessageLog is shared:

  @Data
  @Accessors(chain = true)
  public class MessageLog implements Serializable {
      private String msgId;
      /**
       * Sender system name {@ link io.wilson.common.message.constant.MessageConstant}
       */
      private String system;
      /**
       * Message object json string
       */
      private String msgContent;
      /**
       * Business execution results
       */
      private Boolean success;
      private LocalDateTime createTime;
      private LocalDateTime updateTime;
  
      /**
       * Initialize message logging
       *
       * @param message       news
       * @return
       */
      public static <T extends BaseMessage> MessageLog convertFromMessage(T message) {
          LocalDateTime now = LocalDateTime.now();
          return new MessageLog()
                  .setSystem(message.getSystem())
                  .setSuccess(false)
                  .setCreateTime(now)
                  .setUpdateTime(now);
      }
  }
  
  

  The core points of personal consideration in the design and development of the consumption process are as follows:

  1. If the ordinary message class (such as SmsMessage) is used as the mapping object stored in db, unnecessary attributes (such as createTime, updateTime, success) will be mixed in the message class, and as a general message data carrier, the ordinary message class is more suitable to be used as a VO rather than a DO. Therefore, the processing result of the message and the creation and update time of the message will be used as the attachment on the original message Content is more suitable for maintenance in other database mapping objects, so MessageLog is defined as the entity class of message record
  2. Since it is a general message that can be used by all applications, there must be a certain amount of data. Although the mapping entities are the same, it can improve the convenience of operation and get better performance if stored in different collection s. System coding can better filter messages according to the system
  3. In the message consumption process, only the database name is different in the two steps of ensuring message idempotence and message log entry. Therefore, a parent Listener can be defined to abstract the method of message monitoring consumption. The business processing of different messages is handed over to different message services. The consumption of the same type of message may be subdivided and different message business methods may be called for consumption (such as sending a single message Send SMS in batches), so a consumption () method can be abstracted for each service to call specific service business methods for message consumption according to parameters

  • Message center class diagram and consumption flow chart

    To better show the relationship between classes in the message center, draw the following class diagram:
    

    When a SMS message is sent to the message center, its consumption flow is as follows:
    

  • Message business processing code

    BaseMessageService: abstract interface of message business consumption, which abstracts the business consumption method called by each listener

    public interface BaseMessageService<T extends BaseMessage> {
    
        /**
         * Consumer News
         *
         * @param message         news
         * @param consumeFunction Consumption method
         */
        default boolean consume(T message, Function<T, Boolean> consumeFunction) {
            return consumeFunction.apply(message);
        }
    }
    

    BaseMessageService: abstract interface of SMS business

    @Service
    public interface SmsMessageService extends BaseMessageService<SmsMessage> {
    
        /**
         * Send a single SMS message
         *
         * @param smsMessage
         * @return Business processing results
         */
        boolean sendSingle(SmsMessage smsMessage);
    
    }
    

    SmsMessageServiceImpl: SMS business implementation class

    @Service
    @Slf4j
    public class SmsMessageServiceImpl implements SmsMessageService {
    
        @Override
        public boolean sendSingle(SmsMessage smsMessage) {
            // SMS operation result
            boolean isSuccess = true;
            /*
             * SMS operation and set the operation result to isSuccess
             */
            if (Objects.equals(smsMessage.getToUserId(), "Wilson")) {
                isSuccess = false;
                log.info("Failed to send SMS,Message content:{}", smsMessage);
            }
            return isSuccess;
        }
    }
    

  • Message business processing code

    MessageLogConstant: maintain the related constants of MessageLog (such as the collection name of different messages)

    public interface MessageLogConstant {
    
        /**
         * Mongo collection name of each message log
         */
        interface CollectionName {
            String SMS = "sms_message_log";
            String MAIL = "mail_message_log";
        }
    
    }
    

    AbstractMQStoreListener: in the abstract Listener class method to ensure message idempotence and message log entry operation

    @Slf4j
    public abstract class AbstractMQStoreListener {
    
        @Resource
        protected MongoTemplate mongoTemplate;
    
        /**
         * Determine whether the message has been consumed
         *
         * @param msgId
         * @return
         */
        protected boolean isConsumed(String msgId) {
            long count = mongoTemplate.count(new Query(Criteria.where("msg_id").is(msgId)), collection());
            if (count > 0) {
                log.info("news{}It has been consumed successfully. Please do not send it again!", msgId);
                return true;
            }
            return false;
        }
    
        /**
         * mongo collection name of current message: {@ link io.wilson.message.domain.constant.MessageLogConstant.CollectionName}
         *
         * @return collection name of the current message store
         */
        protected abstract String collection();
    
        /**
         * Save message consumption record
         *
         * @param success Business execution results
         * @param msgId   Message id
         * @param message
         */
        void store(boolean success, String msgId, BaseMessage message) {
            MessageLog messageLog = MessageLog.convertFromMessage(message)
                    .setMsgId(msgId)
                    .setMsgContent(JSONObject.toJSONString(message))
                    .setSuccess(success);
            mongoTemplate.insert(messageLog, collection());
        }
    }
    

    SmsMessageListener: SMS message listener (consumer). If an exception is thrown during the consumption process, RocketMQ will resend the message for consumption at a certain time interval

    @Slf4j
    @Service
    @ConditionalOnProperty(MessageConstant.Topic.SMS_TOPIC)
    @RocketMQMessageListener(topic = MessageConstant.Topic.SMS_TOPIC_TEMPLATE, consumerGroup = MessageConstant.Consumer.SMS_GROUP_TEMPLATE)
    public class SmsMessageListener extends AbstractMQStoreListener implements RocketMQListener<MessageExt> {
        @Resource
        private SmsMessageService smsMessageService;
        private static final String EXCEPTION_FORMAT = "Message consumption failed, message content:%s";
    
        @Override
        public void onMessage(MessageExt message) {
            String msgId = message.getMsgId();
            if (isConsumed(msgId)) {
                return;
            }
            SmsMessage smsMessage = JSONObject.parseObject(message.getBody(), SmsMessage.class);
            log.info("Message received{}: {}", msgId, smsMessage);
            /*if (Objects.equals(smsMessage.getToUserId(), "2020")) {
                log.error("Message {} consumption failed ", message.getMsgId());
                // Throw an exception to let RocketMQ resend the message and consume it again
                throw new MQConsumeException(String.format(EXCEPTION_FORMAT, smsMessage));
            }*/
            boolean isSuccess = smsMessageService.consume(smsMessage, smsMessageService::sendSingle);
            if (!isSuccess) {
                log.info("SMS business operation failed,news id: {}", msgId);
            }
            // Save message consumption record
            store(isSuccess, msgId, smsMessage);
        }
    
        @Override
        protected String collection() {
            return MessageLogConstant.CollectionName.SMS;
        }
    }
    

    MessageCenterApplication: main program

    @SpringBootApplication
    @EnableDiscoveryClient
    public class MessageCenterApplication {
        public static void main(String[] args) {
            SpringApplication.run(MessageCenterApplication.class, args);
        }
    }
    

    Spring Cloud configuration file bootstrap.yml

    eureka:
      client:
        service-url:
          defaultZone: http://localhost:8000/eureka
    spring:
      cloud:
        config:
          discovery:
            enabled: true
            service-id: config-center
          #     Resource file name
          profile: dev
          name: rocketmq
    

    SmsSendTest: unit test class

    @SpringBootTest(classes = MessageCenterApplication.class)
    @RunWith(SpringJUnit4ClassRunner.class)
    public class SmsSendTest {
        @Resource
        private RocketMQTemplate rocketMQTemplate;
        @Value(MessageConstant.Topic.SMS_TOPIC_TEMPLATE)
        private String smsTopic;
    
        @Test
        public void sendSms() {
            SmsMessage smsMessage = new SmsMessage();
            smsMessage.setToUserId("13211")
                    .setMobile("173333222")
                    .setContent("Test SMS messages")
                    .setSystem(MessageConstant.System.QUESTION);
            rocketMQTemplate.send(smsTopic, MessageBuilder.withPayload(smsMessage).build());
        }
    }
    

• Configuration center (config server)

  Main program ConfigServerApplication

  @SpringBootApplication
  @EnableDiscoveryClient
  @EnableConfigServer
  public class ConfigServerApplication {
     public static void main(String[] args) {
        SpringApplication.run(ConfigServerApplication.class, args);
     }
  }
  

  Spring Cloud configuration file bootstrap.yml:

  spring:
    cloud:
      config:
        server:
          git:
            uri: https://gitee.com/Wilson-He/rocketmq-message-center-demo.git
            username: Wilson-He
            force-pull: true
            password:
            # Directory of configuration file under uri
            search-paths: /config-server-properties
  eureka:
    client:
      service-url:
        defaultZone: http://localhost:8000/eureka
  

  Configuration file configs-server-properties/rocketmq-dev.properties:

  rocketmq.name-server=127.0.0.1:9876
  rocketmq.topic.sms=sms-topic
  rocketmq.producer.group.sms=sms-group
  rocketmq.consumer.group.sms=sms-group
  rocketmq.topic.mail=mail-topic
  rocketmq.producer.group.mail=mail-group
  rocketmq.consumer.group.mail=mail-group
  

Operation process

1. Run rocketmq name server and broker, such as mqnamesrv -n 127.0.0.1:9876,mqbroker -n 127.0.0.1:9876
2. Running the eureka application
3. Run configuration center config server
4. Run message center
5. Run the message center unit test class (SmsSendTest) or run the question app to visit localhost:8080/question/toUser?userId=xxx for consumption test. The message center console prints out the log information and Mongo SMS? Message? Log, and successfully adds data, that is, the project is completed
   
   

(to be) extension point:

1. The sender application of rocketmq can set the rocketmq.producer.retry-times-when-send-failed/retry-times-when-send-async-failed property in the configuration file to configure the number of retries after rocketmq fails to send messages synchronously or asynchronously. Otherwise, the default value is 2
2. When the business execution result fails, the reason why it is still received is that sometimes the business execution process may include the operation of calling the third party. When the third party reports an error, the business operation result will fail, and the operation of the third party is uncontrollable. Therefore, the error result should be saved for traceability first, and the business can be re executed through the scheduled task library check if necessary
3. In this example, only one message configuration file is used. In actual development, the message configuration needs to be configured to the corresponding project configuration file according to the needs of the project. For example, the message configuration of question app (such as topc, producer group) should be configured in the configuration file of its project (such as application.yml, apollo's namespace)
4. The NameServer and Broker in this project are not deployed in a cluster. After the Broker cluster deployment, synchronous dual write is configured to avoid the situation that messages are lost because the host has not been synchronized to the slave after writing (intended self Baidu: RocketMQ synchronous dual write)

end

This article demonstrates some ways to use Spring Boot RocketMQ to deal with MQ common problems through a simple project example:

• The problem of message repeated consumption can be guaranteed idempotence through database storage
• If the message consumption business operation fails, you can throw an exception through the Listener to ask RocketMQ to resend the message for consumption

Project source code