First, what is MQ? What is the principle of MQ?
MQ is the message queue, short for Message Queue. Message queuing is a way of communication. The essence of a message is a data structure. Because MQ centralizes the processing and storage of messages in the project, MQ has the functions of decoupling, concurrency and peak shaving.
1. Decoupling:
MQ message producers and consumers do not care about each other's existence. Through the existence of MQ middleware, the whole system achieves decoupling effect.
If RPC is used to communicate between services, when a service communicates with hundreds of services, if the communication interface of that service changes, then the communication interface of hundreds of services changes, which is a very headache.
But after adopting MQ, both producers and consumers can change themselves independently. Their changes will not affect other services. So the decoupling can be achieved. Why decouple? To put it plainly, it is convenient to reduce unnecessary workload.
2, concurrent
MQ has producer clusters and consumer clusters, so when the client is a billion-level user, they are all parallel. Thus, the response speed is greatly improved.
3, peak clipping.
Because MQ can store a large amount of messages, it can store a large number of message requests first, and then slowly process them concurrently.
If RPC communication is used, every request calls RPC interface. When the amount of requests is huge, because RPC requests are very resource-intensive, huge requests will overwhelm the server.
The aim of peak shaving is to improve the user experience and stabilize the whole system. Can withstand a large number of request messages.
2. What MQ is on the market now?
RocketMQ
There are many MQ products on the market, such as RabbitMQ, ActiveMQ, ZeroMQ, RocketMQ, Kafka and so on. These are all open source MQ products. RabbitMQ has been recommended by many people in the past. It is also a very useful MQ product. I will not introduce it too much here. Kafka is also the leader of high throughput, and we will not introduce it here.
We will highlight RocketMQ, Alibaba's open source distributed messaging middleware in 2012, which has been donated to the Apache Software Foundation and became Apache's top project on September 25, 2017.
As a domestic middleware which has experienced the baptism of Alibaba Shuangxi Super Project many times and has stable and outstanding performance, it has been used by more and more domestic enterprises in recent years because of its high performance, low latency and high reliability.
Functional overview
You can see that RocketMQ supports timing and delay messages, which RabbitMQ does not have.
****** Physical Structure of RocketMQ****
It can be seen from this that RocketMQ involves four clusters: producer, Name Server, Consumer and Broker.
Producer cluster:
RocketMQ provides three ways to send messages: synchronous, asynchronous and one-way.
First, general news
1. Synchronization schematic
****** Synchronized Message Key Code******
try { SendResult sendResult = producer.send(msg); // Sending messages synchronously is successful as long as no exception is thrown if (sendResult != null) { System.out.println(new Date() + " Send mq message success. Topic is:" + msg.getTopic() + " msgId is: " + sendResult.getMessageId()); } catch (Exception e) { System.out.println(new Date() + " Send mq message failed. Topic is:" + msg.getTopic()); e.printStackTrace(); } }
****** 2, asynchronous schematic ****
****** Asynchronous Message Key Code****
producer.sendAsync(msg, new SendCallback() { @Override public void onSuccess(final SendResult sendResult) { // Successful Consumer Delivery System.out.println("send message success. topic=" + sendResult.getTopic() + ", msgId=" + sendResult.getMessageId()); } @Override public void onException(OnExceptionContext context) { System.out.println("send message failed. topic=" + context.getTopic() + ", msgId=" + context.getMessageId()); } });
****** 3, One Way Transmission Schema****
One-way only send, do not wait for return, so the fastest speed, generally in microsecond level, but may be lost
One way key code for sending messages
producer.sendOneway(msg);
2. Timing and Delay Messages
Send Timing Message Key Code
try { // Timing messages, units of milliseconds (ms), are delivered after a specified timestamp (current time), such as 2016-03-07 16:21:00. If set to a time before the current timestamp, the message will be delivered to the consumer immediately. long timeStamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2016-03-07 16:21:00").getTime(); msg.setStartDeliverTime(timeStamp); // Sending a message without throwing an exception is successful SendResult sendResult = producer.send(msg); System.out.println("MessageId:"+sendResult.getMessageId()); } catch (Exception e) { // Message sending fails, requiring retry processing, which can be reissued or persisted to compensate for this data processing System.out.println(new Date() + " Send mq message failed. Topic is:" + msg.getTopic()); e.printStackTrace(); }
****** Delay message key code******
try { // Delayed messages, in milliseconds (ms), are delivered at a specified delay time (after the current time), such as messages delivered after three seconds long delayTime = System.currentTimeMillis() + 3000; // Set the time msg.setStartDeliverTime(delayTime) when the message needs to be delivered; SendResult sendResult = producer.send(msg); // Sending messages synchronously is successful as long as no exception is thrown if (sendResult != null) { System.out.println(new Date() + " Send mq message success. Topic is:" + msg.getTopic() + " msgId is: " + sendResult.getMessageId()); } } catch (Exception e) { // Message sending fails, requiring retry processing, which can be reissued or persisted to compensate for this data processing System.out.println(new Date() + " Send mq message failed. Topic is:" + msg.getTopic()); e.printStackTrace(); }
Matters needing attention
1. The msg.setStartDeliverTime parameter for timing and delay messages needs to be set to a time (in milliseconds) after the current timestamp. If set to a time before the current timestamp, the message will be delivered to the consumer immediately.
2. The msg.setStartDeliverTime parameter for timing and delay messages can be set at any time (in milliseconds) within 40 days. Message delivery over 40 days will fail.
3. Start DeliverTime is the time when the server starts delivering to the consumer. If the consumer currently has a message stack, then the timing and delay messages will be placed behind the stacked messages, which will not be delivered strictly according to the configuration time.
4. Because there may be time difference between client and server, there may be deviation between the actual delivery time of message and the delivery time set by client.
5. After setting the delivery time of timed and delayed messages, it is still limited by the three-day message storage time. For example, a timed message can be consumed after five days, and if it is not consumed after the fifth day, the message will be deleted on the eighth day.
6. Delay messages are not supported in other languages except Java.
Publishing message schematics
3. Transaction news
RocketMQ provides distributed transaction functions similar to X/Open XA to ensure the eventual consistency of business senders and MQ messages. The essence of RocketMQ is to place distributed transactions on MQ side in a semi-messaging manner.
Schematic diagram
Among them:
1. Send messages to the RocketMQ server of the message queue.
2. After the server persists the message successfully, it confirms to the sender ACK that the message has been sent successfully, and then the message is half-message.
3. The sender begins to execute the local transaction logic.
4. The sender submits a second confirmation (Commit or Rollback) to the server according to the results of local transaction execution. When the server receives the Commit status, it marks the semi-message as deliverable, and the subscriber will eventually receive the message. When the server receives the Rollback status, it deletes the semi-message, and the subscriber will not accept the message.
5. In the special case of disconnection or application restart, the second confirmation submitted in step 4 above did not reach the server. After a fixed period of time, the server will initiate a message review.
6. When the sender receives the message for review, it needs to check the final result of the local transaction execution of the corresponding message.
7. The sender submits a second confirmation according to the final state of the local transaction checked, and the server still operates on the half-message in step 4.
Notes for RocketMQ's semi-messaging mechanism are
1. According to step 6, we can see that he requires the sender to provide a business review interface.
2. It is not guaranteed that the sender's message is idempotent, and duplicate messages may exist if ack does not return.
3. Consumers should be idempotent.
Core code
Final Business Service Business Service = new Business Service ();//local business
TransactionProducer producer = ONSFactory.createTransactionProducer(properties, new LocalTransactionCheckerImpl()); producer.start(); Message msg = new Message("Topic", "TagA", "Hello MQ transaction===".getBytes()); try { SendResult sendResult = producer.send(msg, new LocalTransactionExecuter() { @Override public TransactionStatus execute(Message msg, Object arg) { // Message ID (it is possible that the message body is the same, but the message ID is different, and the current message ID cannot be queried at the console) String msgId = msg.getMsgID(); // Message body content for crc32, can also use other such as MD5 long crc32Id = HashUtil.crc32Code(msg.getBody()); // Message ID and crc32id are mainly used to prevent message duplication // If the business itself is idempotent, we can ignore it, otherwise we need to use msgId or crc32Id to do idempotent. // If the message is required to be absolutely non-repetitive, it is recommended to use crc32 or MD5 for the body of the message to prevent duplication of messages. Object businessServiceArgs = new Object(); TransactionStatus transactionStatus =TransactionStatus.Unknow; try { boolean isCommit = businessService.execbusinessService(businessServiceArgs); if (isCommit) { // When the local transaction succeeds, the message transactionStatus = TransactionStatus.CommitTransaction is submitted. } else { // If the local transaction fails, the message transactionStatus = TransactionStatus.RollbackTransaction is rolled back. } } catch (Exception e) {log.error("Message Id:{}", msgId, e); } System.out.println(msg.getMsgID());log.warn("Message Id:{}transactionStatus:{}", msgId, transactionStatus.name()); return transactionStatus; } }, null); } catch (Exception e) { // Message sending fails, requiring retry processing, which can be reissued or persisted to compensate for this data processing System.out.println(new Date() + " Send mq message failed. Topic is:" + msg.getTopic()); e.printStackTrace(); }
****** Principle Diagram of All Information Release****
producer is completely stateless and can be deployed in clusters.
Name Server Cluster:
NameServer is an almost stateless node, which can be deployed in a cluster without any information synchronization between nodes. NameServer is very similar to the function of registry.
I heard that Name Server was made by Zoo Keeper before Ali. Maybe because Zookeeper can't meet the requirement of large-scale concurrency, Name Server was developed by Ali.
Name Server is actually a routing table that manages discovery and registration between Producer and Comsumer.
Broker Cluster:
Broker deployment is relatively complex. Broker is divided into Master and Slave. A Master can correspond to multiple Slavers, but a Slaver can only correspond to one Master. Master and Slaver correspond by specifying the same BrokerName.
Different BrokerId is defined. BrokerId is 0 for Master and non-zero for Slaver. Master can deploy multiple. Each Broker establishes a long connection with all nodes in the NameServer cluster and registers Topic information to all NameServers on a regular basis.
Consumer Cluster:
Subscription mode
Message queue RocketMQ supports the following two subscriptions:
Cluster Subscription: All Consumers identified by the same Group ID share consumption messages on average. For example, if a Topic has nine messages and a Group ID has three Consumer instances, then in the cluster consumption mode, each instance is allocated equally and only three of them are consumed.
// Cluster Subscription Settings (Cluster Subscription by default if not set) properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.CLUSTERING);
Broadcast subscription: All Consumers identified by the same Group ID consume a message one time. For example, if a Topic has nine messages and a Group ID has three Consumer instances, then each instance will consume nine messages in the broadcast consumption mode.
// Broadcast subscription settings properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.BROADCASTING);
****** Subscription Message Key Code: ********
Consumer consumer = ONSFactory.createConsumer(properties); consumer.subscribe("TopicTestMQ", "TagA||TagB", **new** MessageListener() { //Subscribe to multiple Tag s public Action consume(Message message, ConsumeContext context) { System.out.println("Receive: " + message); return Action.CommitMessage; } }); //Subscribe to another Topic consumer.subscribe("TopicTestMQ-Other", "*", **new** MessageListener() { //Subscribe to all Tag s public Action consume(Message message, ConsumeContext context) { System.out.println("Receive: " + message); return Action.CommitMessage; } }); consumer.start();
Matters needing attention:
Consumer end should do idempotent processing, all MQ will not do idempotent processing basically, need business end processing, because if doing idempotent processing at MQ end will bring the complexity of MQ, and seriously affect the performance of MQ.
Message Receiving and Sending Model
Master and Subaccount Creation
The reason for creating master and child accounts is permission issues. Here is the main account creation flow chart
Subaccount flow chart
MQ is a micro-service architecture
A very important part
The birth of MQ provides a way to change the original synchronous architecture thinking to asynchronous architecture thinking, which provides a good solution for the stable implementation of large-scale, high concurrent business scenarios.
Martin Fowler emphasizes that the first principle of distributed invocation is not to be distributed. This sentence seems quite philosophical, but as far as enterprise application systems are concerned, this principle will be broken as long as the whole system keeps evolving and multiple subsystems coexist.
Martin Fowler put forward this principle. On the one hand, he hoped that the designer could treat the distributed call prudently, on the other hand, it was also caused by the defects of the distributed system itself.
So micro-service is not a panacea, the best architecture is the appropriate architecture.