1, Concept
1. Advantages
Application decoupling: improving system fault tolerance and maintainability
Asynchronous acceleration: improve user experience and system throughput
Peak shaving and valley filling: improving system stability
2. Disadvantages
Reduced system availability
The more external dependencies the system introduces, the worse the stability of the system. Once MQ goes down, it will have an impact on the business.
- How to ensure high availability of MQ? Cluster mode
The complexity of the system is improved
The addition of MQ greatly increases the complexity of the system. In the past, synchronous remote calls were made between systems, but now asynchronous calls are made through MQ.
- How to ensure that messages are not consumed repeatedly? Message confirmation mechanism of ACK
- How to deal with message loss? Message confirmation mechanism of ACK
- How to ensure the order of message delivery? It can be handled in a roundabout way, and multiple consumers can be kneaded into one. Polling mechanism is adopted for processing
ACK message confirmation mechanism.
A: the ACK mechanism is that after the consumer receives a message from RabbitMQ and processes it, it feeds back to RabbitMQ, and RabbitMQ deletes the message from the queue after receiving the feedback.
Consistency problem
After processing the business, system A sends message data to systems B, C and D through MQ. If system B and C process successfully, system D process fails.
- How to ensure the consistency of message data processing?
2, Common MQ products
3, Working mode
1, Simple mode
A producer, a consumer
2, Work mode
In the work queue mode, the default switch is used, and the routing name is the queue name. When multiple terminals consume the same queue, the switch uses polling to send messages. Generally speaking, it sends one message to the first and the next
Application scenario:
When the task is too heavy or there are many tasks, using work queue can improve the speed of task processing.
Test:
1. Multiple consumers.
2. The producer sends multiple messages.
result:
1. A message will only be received by one consumer;
2. rabbit uses polling to send messages to consumers on average;
3. Consumers will not receive the next message until they have processed a message.
3, Publish/subscribe subscription mode
Publish subscribe mode:
1. Each consumer listens to their own queue.
2. The producer sends the message to the switch, and the switch forwards the message to each queue bound to the switch, and each queue bound to the switch will receive the message
Application scenario:
User notification: when the user recharge is successful or the transfer is completed, the system notifies the user. The notification methods include SMS and email.
technological process:
1. Producers
Declare a fanout type switch.
Declare two queues and bind to this switch. You do not need to specify routingkey when binding
You do not need to specify routingkey when sending messages
2. Consumers
The exchange opportunity publishes the information to each queue listening to the switch
4, Routing mode
Routing mode:
1. Each consumer listens to their own queue and sets the routing key.
2. The producer sends the message to the switch, and the switch forwards the message to the specified queue according to the routingkey.
technological process:
1. Producers
Declare a DIRECT type switch.
Declare two queues and bind to this switch. routingkey needs to be specified when binding
routingkey needs to be specified when sending messages
2. Consumers
When binding the queue, the consumer can specify the routingkey to obtain only the messages with the specified routingkey
explain:
The Routing mode requires the queue to specify the routingkey when binding the switch, and the message will be forwarded to the queue that meets the routingkey.
5, Topics wildcard pattern
It is similar to the routing mode, but the matching of routingkey is determined by wildcard. Only when the routing modes are equal can they be matched
Set the switch type to Topics
Application scenario:
Notify the user according to the user's notification settings. Set the user receiving Email to only receive Email, and set the user receiving sms to only receive sms. Set two types
If both notification types are received, both notifications are valid.
Wildcard use:
[#], match one or more words, such as uncle. #, which can match uncle.sms, uncle.email and uncle.sms.email
[*], match a word, such as uncle. *, which can match uncle.sms and uncle.email
example:
uncle.#.sms. # can match uncle.sms and uncle.email.sms, but cannot match uncle.email
6, RPC
RPC is the method by which the client remotely calls the server. The asynchronous call of RPC can be realized by using MQ, which is based on the Direct switch. The process is as follows:
1. The client is the producer and consumer. It sends RPC call messages to the RPC request queue and listens to the RPC response queue.
2. The server listens to the message of the RPC request queue. After receiving the message, it executes the server's method to get the result returned by the method
3. The server sends the result of the RPC method to the RPC response queue
4. The client (RPC caller) listens to the RPC response queue and receives the RPC call result.
4, Dead letter queue
When the message becomes Dead Message, it can be re sent to another switch, which is the dead letter switch DLX
There are three situations when a message becomes a dead letter
-
Queue message length reached limit
-
The consumer rejects the consumption message and does not return to the queue
-
The message expiration setting exists in the original queue, and the message arrival timeout has not been consumed
Usage scenario
-
TTL + dead letter queue combination can be used to achieve the effect of delay queue
-
You can set the dead letter queue in the dead letter queue
5, Configuration items in springboot
Producer -- > message -- > switch -- > queue -- > becomes dead letter -- > Dlx switch -- > queue -- > consumer
1, Connection configuration
#Container type. simple or direct spring.rabbitmq.listener.type=simple #This configuration item is used to indicate the message confirmation method. There are three configuration methods: none, manual and auto. #none means that no reply will be sent. #manual means that the listener must notify all messages by calling Channel.basicAck(). #auto means that the container will respond automatically unless the MessageListener throws an exception, which is the default configuration. spring.rabbitmq.listener.simple.acknowledge-mode=manual #This configuration item determines whether the messages rejected due to the exception thrown by the listener are put back on the queue. The default value is true. #If this property is configured as true, it will be put back to the queue. If it is configured as false, it will not be put back to the queue. spring.rabbitmq.listener.simple.default-requeue-rejected=false #The NONE value disables the publish confirmation mode and is the default value #The CORRELATED value is the callback method that will be triggered after the message is successfully published to the exchange, as shown in example 1 #After testing, the SIMPLE value has two effects. One effect is the same as the CORRELATED value, which will trigger the callback method. The other is to use the rabbitTemplate to call the waitForConfirms or waitForConfirmsOrDie method after successfully publishing the message, wait for the broker node to return the sending result, and determine the logic of the next step according to the returned result, Note that if the waitForConfirmsOrDie method returns false, the channel will be closed, and messages cannot be sent to the broker next; spring.rabbitmq.publisher-confirm-type=correlated #This ensures that messages are returned when they are not received by the queue spring.rabbitmq.publisher-returns=true #Specifies whether the message is forcibly returned or directly discarded when it is not received by the queue spring.rabbitmq.template.mandatory=true #The maximum number of nack messages that can be processed by a consumer must be greater than or equal to the number of transactions if there are transactions #Maximum number of nack messages per consumer spring.rabbitmq.listener.direct.prefetch=1000
2, Producer message reliability delivery
1. Confirmation mode
//1. Turn on the confirmation mode
//2. Define the ConfirmCallback callback callback function in rabbitTemplate
rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
/**
* @param correlationData Related configuration information
* @param b Whether the switch successfully receives the message. true succeeds and false fails
* @param s Failure reason
*/
@Overr
@Override
public void confirm(CorrelationData correlationData, boolean b, String s) {
}
});
2. Fallback mode
//Fallback mode: when the message is sent to the switch and the switch fails to route to the queue, it will be executed. ReturnCallBack
//step
//1. Enable fallback mode
//2. Set the ReturnCallBack function
//3. Set the message processing mode of the switch
// 1. If the message is not routed to the queue, the message is discarded (default)
// 2. If the message is not routed to the queue, it is returned to the message sender ReturnCallBack
//Sets the mode in which the switch processes failure messages
rabbitTemplate.setMandatory(true);
rabbitTemplate.setReturnsCallback(new RabbitTemplate.ReturnsCallback() {
@Override
public void returnedMessage(ReturnedMessage returned) {
}
});
3, Consumer message reliability delivery
1. Automatic sign in
2. Manual sign in
//Refuse to sign //The second boolean indicates whether to return to the queue. If true, the message returns to the queue, and the broker will resend the message to the consumer channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false); //Normal manual sign in //Whether to sign in multiple messages. true is yes channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
4, TTL expiration time
If all are set, it will take effect in a short time
1. Unified queue expiration
//This parameter is configured in milliseconds when the queue is declared
args.put("x-message-ttl",100000);
2. The message expires separately
For messages that expire individually, whether they expire or not is determined only at the top of the queue
//The message post-processing object sets some message parameter information
MessagePostProcessor messagePostProcessor=new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
//1. Set the configuration information of message
//Expiration time in milliseconds
message.getMessageProperties().setExpiration("5000");
return message;
}
};
rabbitTemplate.convertSendAndReceive(BUSINESS_EXCHANGE_NAME, "", msg,messagePostProcessor);
5, springboot producer configuration
1. Declare service switch
2. Declare dead letter switch
3. Declare dead letter queue
4. Declare service queue
- Set the dead letter switch of the current queue
- Set the dead letter routing KEY of the current queue
5. Bind service queue and switch relationship
6. Bind the relationship between dead letter queue and dead letter switch
// Declare business Exchange
@Bean("businessExchange")
public FanoutExchange businessExchange(){
return new FanoutExchange(BUSINESS_EXCHANGE_NAME);
}
// Declare service queue A
@Bean("businessQueueA")
public Queue businessQueueA(){
Map<String, Object> args = new HashMap<>(2);
// x-dead-letter-exchange here declares the dead letter switch bound to the current queue
args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
// x-dead-letter-routing-key here, the dead letter routing key of the current queue is declared
args.put("x-dead-letter-routing-key", DEAD_LETTER_QUEUEA_ROUTING_KEY);
//Set queue expiration time
args.put("x-message-ttl",100000);
//Set the queue length to 10 messages
args.put("x-max-length",10);
return QueueBuilder.durable(BUSINESS_QUEUEA_NAME).withArguments(args).build();
}
// Declare dead letter queue A
@Bean("deadLetterQueueA")
public Queue deadLetterQueueA(){
return new Queue(DEAD_LETTER_QUEUEA_NAME);
}
// Declare business queue A binding relationship
@Bean
public Binding businessBindingA(@Qualifier("businessQueueA") Queue queue,
@Qualifier("businessExchange") FanoutExchange exchange){
return BindingBuilder.bind(queue).to(exchange);
}
// Declare the binding relationship of dead letter queue A
@Bean
public Binding deadLetterBindingA(@Qualifier("deadLetterQueueA") Queue queue,
@Qualifier("deadLetterExchange") DirectExchange exchange){
return BindingBuilder.bind(queue).to(exchange).with(DEAD_LETTER_QUEUEA_ROUTING_KEY);
}
6, springboot consumer configuration
//Set listening queue
@RabbitListener(queues = BUSINESS_QUEUEA_NAME)
public void receiveA(Message message, Channel channel) throws IOException {
String msg = new String(message.getBody());
log.info("Business message received A: {}", msg);
boolean ack = true;
Exception exception = null;
try {
if (msg.contains("deadletter")){
throw new RuntimeException("dead letter exception");
}
} catch (Exception e){
ack = false;
exception = e;
}
if (!ack){
log.error("Message consumption is abnormal, error msg:{}", exception.getMessage(), exception);
//Refuse to sign basicNack
//The second boolean indicates whether to return to the queue. If true, the message returns to the queue, and the broker will resend the message to the consumer
channel.basicNack(message.getMessageProperties().getDeliveryTag(), false, false);
} else {
//Manual normal sign in basicAck
//Whether to sign in multiple messages. true is yes
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}
}
7, Application problems
1, Message compensation (reliability)
- The consumer receives the message and calls the producer's API after processing.
2, Idempotency
Consume multiple identical messages and get the same result as consuming the message once
In the database, optimistic locking mechanism is used
liveryTag(), false, false);
} else {
//Manual normal sign in basicAck
//Whether to sign in multiple messages. true is yes
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
}
}
# 7, Application problems ## 1, Message compensation (reliability) - The consumer receives the message and calls the producer after processing. API [External chain picture transfer...(img-uevDGpwY-1634552488751)] ## 2, Idempotency Consume multiple identical messages and get the same result as consuming the message once In the database, optimistic locking mechanism is used