Springboot integration RabbitMQ details

RabbitMQ

This paper combines some online materials for reference. If there is infringement, please point out.

Characteristics of RabbitMQ

RabbitMQ is an open source message middleware developed in Erlang language to implement AMQP (Advanced message queuing protocol). First, you should know some characteristics of RabbitMQ, Official website Available:

Reliability. Support persistence, transmission confirmation, release confirmation, etc. to ensure the reliability of MQ.
Flexible message distribution strategy. This should be a major feature of RabbitMQ. The exchange (switch) routes messages before they enter MQ.
Message distribution strategies include: simple mode, work queue mode, publish subscribe mode, routing mode and wildcard mode.
Support cluster. Multiple RabbitMQ servers can form a cluster to form a logical Broker.
Multiple protocols. RabbitMQ supports a variety of message queuing protocols, such as STOMP, MQTT, and so on.
Support multiple language clients. RabbitMQ supports almost all common programming languages, including Java,. NET, Ruby, and so on.
Visual management interface. RabbitMQ provides an easy-to-use user interface that allows users to monitor and manage message brokers.
Plug in mechanism. RabbitMQ provides many plug-ins, which can be extended through plug-ins or write their own plug-ins.

AMQP

AMQP Model

The message is sent to the exchange by the publisher, and then the exchange distributes the received message to the bound queue according to the routing rules. Finally, the AMQP agent will deliver the message to the consumers who have subscribed to this queue, or the consumers can obtain it by themselves according to their needs.

Message confirmation

From the perspective of security, the network is unreliable, and the application receiving the message may also fail to process the message. For this reason, the AMQP module includes the concept of message acknowledgements: when a message is delivered from the queue to the consumer, the consumer server needs to return an ACK (acknowledgement message), and the broker will delete the message only after receiving the acknowledgement; Message confirmation can be automatic or manual by the consumer. In addition, it also supports the production side to send messages to the broker to get the ack of the broker, so as to respond to the logic.

AMQP is a programmable protocol

In a sense, AMQP entities and routing rules are defined by the application itself, not by the message broker. It includes operations on the protocol itself, such as declaring queues and switches, defining the binding between them, subscribing to queues, and so on. However, attention should be paid to the conflict between the definitions of both parties, otherwise the problem of configuration error will be exposed.

RabbitMQ installation

Windows10 installation

step

To erlang Official website Download the win10 installation package. Install as a fool after downloading.
Configuring erlang environment variables

cmd enter erl to verify whether the installation is successful, as follows:; ctrl+c exit

Install the RabbitMQ service as a fool.
In RabbitMQ gitHub project Download the server installation package of window version
Enter the installation directory, under the sbin directory, execute: rabbitmq plugins enable rabbitmq_ The management command installs the plug-in for the management page

Double click rabbitmq-server.bat startup script or execute RabbitMQ server start in sbin directory, and then open service management to see that RabbitMQ is running.
Open browser input http://localhost:15672 The account password is guest/guest by default

Spring integrates AMQP

Official Chinese documents

Spring AMQP (geekdoc.top)

GitHup translation document

GitHub - Rockit BA / spring rabbit -: spring AMQP implementation: Spring rabbit official Chinese document translation

Main object classes and functions of Spring AMQP

class	effect
Queue	Corresponding to Queue in RabbitMQ
AmqpTemplate	Interface for sending and receiving messages to RabbitMQ
RabbitTemplate	Implementation class of AmqpTemplate
@RabbitListener	Specify the message receiver, which can be configured on classes and methods
@RabbitHandler	Specify the message receiver, which can only be configured on methods and can be used with @ RabbitListener
Message	Encapsulation of RabbitMQ messages
Exchange	Encapsulation of RabbitMQ's Exchange. Subclasses include TopicExchange, FanoutExchange, DirectExchange, etc
Binding	Binding a Queue to an Exchange is only a declaration and does not actually bind
AmqpAdmin	Interface, which is used for Exchange and Queue management, such as create / delete / bind, etc. it automatically checks the Binding class and completes the Binding operation
RabbitAdmin	Implementation class of AmqpAdmin
ConnectionFactory	Create a Connection factory class. RabbitMQ also has a class named ConnectionFactory, but they have no inheritance relationship. Spring ConnectionFactory can be regarded as a encapsulation of RabbitMQ ConnectionFactory
CachingConnectionFactory	The implementation class of Spring ConnectionFactory can be used to cache channels and connections
Connection	Spring is a Connection class used to create channels. RabbitMQ also has a class named Connection, but they have no inheritance relationship. Spring Connection encapsulates RabbitMQ Connection
SimpleConnection	The implementation class of Spring Connection delegates the actual work to the Connection class of RabbitMQ
MessageListenerContainer	Interface. The consumer is responsible for maintaining a connection with the RabbitMQ server and passing the Message to the actual @ RabbitListener/@RabbitHandler for processing
RabbitListenerContainerFactory	Interface for creating MessageListenerContainer
SimpleMessageListenerContainer	Implementation class of MessageListenerContainer
SimpleRabbitListenerContainerFactory	Implementation class of RabbitListenerContainerFactory
RabbitProperties	Property class for configuring Spring AMQP

Main parameters of Spring AMQP

parameter	Default value	explain
Basic information
spring.rabbitmq.host	localhost	host
spring.rabbitmq.port	5672	port
spring.rabbitmq.username	guest	user name
spring.rabbitmq.password	guest	password
spring.rabbitmq.virtual-host		Virtual host
spring.rabbitmq.addresses		server address list (separated by commas). If this item is configured, spring.rabbitmq.host and spring.rabbitmq.port will be ignored
spring.rabbitmq.requested-heartbeat		Request heartbeat timeout, 0 means not specified; If no time is added, the default unit is seconds
spring.rabbitmq.publisher-confirm-type	none	Publish confirmation types, none, correlated, simple. This configuration only determines whether it is posted to the exchange, regardless of whether it is sent to the queue
spring.rabbitmq.publisher-returns	false	Enable publish return
spring.rabbitmq.connection-timeout		Connection timeout, 0 means never timeout
Cache cache
spring.rabbitmq.cache.channel.checkout-timeout		If the channel cache size has been reached, the time to wait to get the channel. If 0, a new channel is always created.
spring.rabbitmq.cache.channel.size		Number of channel s held in the cache
spring.rabbitmq.cache.connection.size		The number of cached connections, which takes effect only in connection mode
spring.rabbitmq.cache.connection.mode	channel	Connection factory cache mode
Listener
spring.rabbitmq.listener.type	simple	Container type, simple or direct
spring.rabbitmq.listener.simple.auto-startup	true	Whether to start the container when the application starts
spring.rabbitmq.listener.simple.acknowledge-mode	auto	Message confirmation methods: none, manual and auto
spring.rabbitmq.listener.simple.concurrency		Minimum number of listener s
spring.rabbitmq.listener.simple.max-concurrency		Maximum number of listener s
spring.rabbitmq.listener.simple.prefetch		The maximum number of nack messages that a consumer can process
spring.rabbitmq.listener.simple.default-requeue-rejected	true	Are rejected messages rejoined
spring.rabbitmq.listener.simple.missing-queues-fatal	true	If the queue declared by the container is unavailable, whether it fails; Or if one or more queues are deleted at run time, stop the container
spring.rabbitmq.listener.simple.idle-event-interval		How often should idle container events be published
spring.rabbitmq.listener.simple.retry.enabled	false	Enable consumer retry
spring.rabbitmq.listener.simple.retry.max-attempts	3	max retries
spring.rabbitmq.listener.simple.retry.max-interval	10000ms	Maximum retry interval
spring.rabbitmq.listener.simple.retry.initial-interval	1000ms	The time interval between the first and second attempts to send a message
spring.rabbitmq.listener.simple.retry.multiplier	1.0	Multiplier applied to the previous retry interval
spring.rabbitmq.listener.simple.retry.stateless	true	Is the retry stateless or stateful
spring.rabbitmq.listener.direct.consumers-per-queue		Number of consumers per queue
Other parameters of direct listener type are the same as those of simple type
Template
spring.rabbitmq.template.mandatory	false	Whether the message is returned when it is not received by the queue is similar to spring.rabbitmq.publisher-returns. This configuration takes precedence over spring.rabbitmq.publisher-returns
spring.rabbitmq.template.receive-timeout		Timeout for receive() operation
spring.rabbitmq.template.reply-timeout		Timeout for sendAndReceive() operation
spring.rabbitmq.template.retry.enabled	false	Send message retry
spring.rabbitmq.template.retry.max-attempts	3.0	Maximum number of retries to send a message
spring.rabbitmq.template.retry.initial-interval	1000ms	The time interval between the first and second attempts to send a message
spring.rabbitmq.template.retry.multiplier	1.0	Multiplier applied to the previous retry interval
spring.rabbitmq.template.retry.max-interval	10000ms	Maximum retry interval

Spring boot integrates AMQP

Comments related to consumer monitoring

@RabbitListener

It can act on a class or method to set the listening queue. If containerFactory() is not set, the default container factory is used.

Many built-in properties provide the relationship of binding queue.

Function on method: indicates that the method listens to a queue
Function on class: @ RabbitHandler needs to be used together. The listening queue will call the method annotated by @ RabbitHandler

@RabbitListener(bindings = @QueueBinding(
    value = @Queue(value = "directQueue-Two", durable = "false"),
    exchange = @Exchange(value = "MqSendService-One", type = "direct", durable = "false"),
    key = "One"),
    ackMode = "MANUAL"
)
public void tsJucDirectMsgTwo(@Header Message data, Channel channel){}

matters needing attention

A listening queue must be specified. It is recommended to specify the binding switch and queue to keep consistent with the production side

**Method 1: * * only declare the listening queue (not recommended)

@RabbitListener(queues = "directQueue-One")

In this way, consumers will listen to this queue by default. If the queue does not exist in the rabbit service broker, errors will always be reported

**Method 2: * * keep synchronization with the production side and specify the binding relationship

@RabbitListener(bindings = @QueueBinding(
    value = @Queue(value = "directQueue-One",type = "direct"),
    exchange = @Exchange(value = "MqSendService-One"),
    key = "One"
))

In this way, if the specified Queue does not exist in the broker, the specified Exchange and Queue will be created directly.

Scenario without queue:

When a message is declared but not sent on the production side, because if it is declared but not sent on the production side, the corresponding Exchange and Queue will not be created.
The queue in the broker is deleted

@RabbitHandler

@The RabbitListener tag on the class indicates that when a message is received, it will be handed over to the method of @ RabbitHandler for processing;

The specific processing method is based on the parameter type after MessageConverter conversion

@Component
@RabbitListener(queues = "consumer_queue")
public class Receiver {
 
    @RabbitHandler
    public void processMessage1(String message) {
        System.out.println(message);
    }
 
    @RabbitHandler
    public void processMessage2(byte[] message) {
        System.out.println(new String(message));
    }
    
}

@Payload

You can get the body information in the message

@RabbitListener(queues = "debug")
public void processMessage1(@Payload String body) {
    System.out.println("body: "+body);
}

@Header，@Headers

You can get the header information in the message

@RabbitListener(queues = "debug")
public void processMessage1(@Payload String body, @Header String token) {
    System.out.println("body: "+body);
    System.out.println("token: "+token);
}

@RabbitListener(queues = "debug")
public void processMessage1(@Payload String body, @Headers Map<String,Object> headers) {
    System.out.println("body: "+body);
    System.out.println("Headers: "+headers);
}

quick get start

Gitee project: Ahang/ts-rabbitmq (gitee.com)

RabbitMQ structure introduction

Queues, switches, and bindings are collectively referred to as AMQP entities

member

ConnectionFactory,Connection

ConnectionFactory, connection and Channel are the most basic objects in the API provided by RabbitMQ. Connection is a TCP connection between publisher / consumer and broker. It encapsulates some logic related to socket protocol. ConnectionFactory is the manufacturing factory of connection.

Channel

If a Connection is established every time RabbitMQ is accessed, the overhead of establishing a TCP Connection when the message volume is large will be huge and the efficiency will be low.

A channel is a logical connection established within a connection. If the application supports multithreading, each thread usually creates a separate channel for communication. The AMQP method contains a channel id to help the client and message broker identify the channel, so the channels are completely separated.

As a lightweight Connection, Channel greatly reduces the overhead of establishing TCP connection by the operating system

Producer

The party producing the message sends the message to the designated switch through the channel;

The producer can declare the Exchange, Queue and corresponding relationship before sending the message; Send a message after declaration. If there is no relevant member, the corresponding Exchange and Queue will be created according to the declaration.

If direct sending is not declared, it will be sent according to the default rules.

Consumer

The party consuming the message consumes the message by listening to the specified queue;

The consumer can also declare the Exchange, Queue and corresponding relationship. After the declaration, if the listener finds that there is no listening Queue, the corresponding Exchange and Queue will be created according to the declaration.

Exchange (switch)

It is used to receive and distribute messages. There are many different types of switches to deal with specific requirements;

Without storage, messages will be stored in the queue; The switch only receives, forwards and distributes messages.

Queue

Messages for storing producers;

RoutingKey

Message routing key rules specified by the producer;

Is to match the bound routing key on the switch to find the queue to send.

//The switch named "topic ex" will match the bound routing key of "One.Two.Three"
rabbitTemplate.send("Topic-Ex","One.Two.Three",msg);

BindingKey

It is used to bind the message of the exchange to the queue;

It specifies the binding routing key of the switch and queue during configuration to match the routing key specified by the producer for sending messages; There is a corresponding binding route between each switch and the queue. First, the message is sent to the specified switch, and then the preset binding route key is matched according to the sent routing rules. Matching the corresponding binding route means that the message finds the corresponding queue.

//The manufacturer specifies the binding relationship through the configuration class
@Bean
public Binding bingExchange2(){
    return BindingBuilder.bind(topicQueue2())   //Bind queue
        .to(topicExchange())       //Which switch is the queue bound to
        .with("*");         //Binding route key, must be specified
}

//The switch and queue relationship of the consumer monitoring statement should be consistent with the above, otherwise a new one will be created
@RabbitListener(bindings = @QueueBinding(
    value = @Queue(value = "topicQueue-One", durable = "false"),
    exchange = @Exchange(value = "Topic-Ex", type = "topic", durable = "false"),
    key = "*"))
public void tsTopicMsg(Message data, Channel channel) {
    String str = new String(data.getBody());
    System.out.println(str + "-----: " + seq);
    seq.incrementAndGet();
}

Virtual host

Each Rabbit can create many vhost s, which we call virtual hosts. In fact, each virtual host is a mini version of Rabbit MQ, with its own queues, switches and bindings, and its own permission mechanism.

Designed for multi tenancy and security reasons, the basic components of AMQP are divided into a virtual Group, similar to the concept of namespace in the network (or RocketMQ's Group).

When multiple different users use the services provided by the same RabbitMQ server, they can be divided into multiple vhosts. Each user creates an exchange / queue in its own vhost.

vhost feature

RabbitMQ's default vhost is "/" out of the box;
Multiple vhosts are isolated, multiple vhosts cannot communicate, and there is no need to worry about naming conflicts (queue, switch and binding), realizing multi-layer separation;
vhost must be specified when creating a user;

vhost operation

You can use the rabbitmqctl tool command

establish

rabbitmqctl add_vhost[vhost_name]

Delete vhost

rabbitmqctl delete_vhost[vhost_name]

View all vhost s

rabbitmqctl list_vhosts

Switch Type

Multi consumer situation

When a queue is monitored by multiple consumers, messages will be evenly distributed to consumers. If a message is blocked, other messages will not be sent to another idle consumer, and the message allocation is fixed at the beginning.

Direct type (default, match send)

It will route the message to those queues whose binding key exactly matches the routing key.

It is a one-to-one model. A message must be sent to a specified queue (exact match).

Configuration code

@Configuration
public class RabbitDirectConfig {

    @Bean
    public Queue directQueue(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("directQueue-One",false,false,false,null);
    }

    @Bean
    public Queue directQueue2(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("directQueue-Two",false,false,false,null);
    }

    @Bean
    public DirectExchange directExchange(){
        //Parameter introduction
        //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
        return new DirectExchange("MqSendService-One",false,false,null);
    }

    @Bean
    public Binding bingExchange(){
        return BindingBuilder.bind(directQueue())   //Bind queue
                .to(directExchange())       //Which switch is the queue bound to
                .with("One");         //Binding route key, must be specified
    }

    @Bean
    public Binding bingExchange2(){
        return BindingBuilder.bind(directQueue2())   //Bind queue
                .to(directExchange())       //Which switch is the queue bound to
                .with("Two");         //Binding route key, must be specified
    }
}

Topic type (extended matching sending)

It is an extension of Direct type and provides flexible matching rules.

routing key is a string separated by a sentence dot "." (we call each independent string separated by a sentence dot "." a word), such as "One.Two"
Like routing key, binding key is also a string separated by sentence dot "."
There can be two special characters "*" and "#" in the binding key for fuzzy matching, of which "*" is used to match one word and "#" is used to match multiple words (which can be zero)

Configuration code

package cn.zh.config;

import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RabbitTopicConfig {
    @Bean
    public Queue topicQueue(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("topicQueue-One",false,false,false,null);
    }

    @Bean
    public Queue topicQueue2(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("topicQueue-Two",false,false,false,null);
    }

    @Bean
    public TopicExchange topicExchange(){
        //Parameter introduction
        //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
        return new TopicExchange("Topic-Ex",false,false,null);
    }

    @Bean
    public Binding bingExchange(){
        return BindingBuilder.bind(topicQueue())   //Bind queue
                .to(topicExchange())       //Which switch is the queue bound to
                .with("*.Two.*");        //Routing key, must be specified
    }

    @Bean
    public Binding bingExchange2(){
        return BindingBuilder.bind(topicQueue2())   //Bind queue
                .to(topicExchange())       //Which switch is the queue bound to
                .with("#"); / / routing key, which must be specified
    }
}

Fanout type (broadcast transmission)

It will route all messages sent to the Exchange to all queues bound to it.

It is a one to many type. The Binding Key cannot be specified. A message sent will be sent to all bound queues.

Configuration code

package cn.zh.config;

import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RabbitFanoutConfig {

    @Bean
    public Queue fanoutQueue(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("fanoutQueue-One",false,false,false,null);
    }

    @Bean
    public Queue fanoutQueue2(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        return new Queue("fanoutQueue-Two",false,false,false,null);
    }

    @Bean
    public FanoutExchange fanoutExchange(){
        //Parameter introduction
        //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
        return new FanoutExchange("Fanout-Ex",false,false,null);
    }

    @Bean
    public Binding bingExchange(){
        return BindingBuilder.bind(fanoutQueue())   //Bind queue
                .to(fanoutExchange());       //Which switch is the queue bound to
    }

    @Bean
    public Binding bingExchange2(){
        return BindingBuilder.bind(fanoutQueue())   //Bind queue
                .to(fanoutExchange());       //Which switch is the queue bound to
    }

}

Headers (key value pairs match, not commonly used)

Header type Exchange does not rely on the matching rules of routing key and binding key to route messages, but matches according to the headers attribute in the sent message content.

Specify a set of key value pairs when binding the Queue to Exchange; when the message is sent to Exchange, RabbitMQ will get the headers of the message (also in the form of a key value pair) and compare whether the key value pairs match the key value pairs specified when binding the Queue to Exchange; if they match exactly, the message will be routed to the Queue, otherwise it will not be routed to the Queue.

This type is not commonly used. No code is provided temporarily.

Message

When performing operations such as basicPublish(), the content is passed as a byte array parameter, while other attributes are passed as separate parameters.

public class Message {

    private final MessageProperties messageProperties;

    private final byte[] body;

    public Message(byte[] body, MessageProperties messageProperties) {
        this.body = body;
        this.messageProperties = messageProperties;
    }

    public byte[] getBody() {
        return this.body;
    }

    public MessageProperties getMessageProperties() {
        return this.messageProperties;
    }
    
    ...
}

The MessageProperties interface defines several common properties, such as "messageId", "timestamp", "contentType", etc. these properties can also be extended by calling setHeader(String key, Object value) method.

Message serialization

Custom classes to be sent as message object s must implement the Serializable interface, otherwise they will receive IllegalArgumentException: SimpleMessageConverter only supports String, byte[] and Serializable payloads.

Starting from version 1.5.7, 1.6.11, 1.7.4, and 2.0.0, if the message body is a serialized Serializable Java object, it will not be deserialized during execution (default), which is to prevent unsafe deserialization. By default, only java.util and java.lang classes are deserialized.

To restore the previous behavior, you can add the allowed class / package pattern Message.addAllowedListPatterns(...) by calling.

//wildcard
Message.addAllowedListPatterns("com.zh.*.class");
//single
Message.addAllowedListPatterns(User.class.getName());

@org.junit.jupiter.api.Test
public void test() {
    NoMessage hello = new NoMessage("hello");
    SimpleMessageConverter simpleMessageConverter = new SimpleMessageConverter();
    Message message = simpleMessageConverter.toMessage(hello, new MessageProperties());
    log.info("Before adding whitelist---{}",message);
    Message.addAllowedListPatterns(NoMessage.class.getName());
    log.info("NoMessage Fully qualified name:{}",NoMessage.class.getName());
    log.info("After adding whitelist---{}",message);
}

Output:
Before adding whitelist---(Body:'[B@6fc3e1a4(byte[89])' MessageProperties
NoMessage Fully qualified name: com.rabbit.producer.NoMessage
 After adding whitelist---(Body:'NoMessage(content=hello)'

Queue

Builder creation

@Bean
public Queue directQueue(){
    //The required attributes can be added continuously by the builder
    Queue queue = QueueBuilder.durable("dis").autoDelete().ttl(100).build();
    return queue;
}

Construction method new

@Bean
public DirectExchange directExchange(){
    Map<String, Object> args = new HashMap<>(3);
    //Declare the dead letter switch bound by the current queue
    args.put("x-dead-letter-exchange", "dead_exchange");
    //Declare the dead letter routing key of the current queue
    args.put("x-dead-letter-routing-key", "dead");
    //Declare the TTL of the queue
    args.put("x-message-ttl", 10000);
    //Parameter introduction
    //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
    return new DirectExchange("MqSendService-One",false,false,args);
}

Characteristic function

Prefetch count (message allocation)

If multiple consumers subscribe to messages in the same Queue at the same time, the messages in the Queue will be shared equally among multiple consumers. This is not good, because if the processing time of each message is different, it may cause some consumers to be busy all the time, while others will soon finish their work and be idle all the time.

We can set prefetchCount to indicate that the consumer can only process several messages in the queue at a time. For example, if we set prefetchCount=1, the consumer can only consume one message in the same queue at a time, and the messages will not be allocated to other messages in the queue before they are processed. In this way, those who can do more can do more.

rabbitmq:
    addresses: 127.0.0.1
    cache:
      channel:
        size: 25
# Specify the consumer message confirmation method
    listener:
      simple:
        # Minimum concurrent number of consumers
        concurrency: 1
        # Maximum concurrent number of consumers
        max-concurrency: 5
        # Number of messages processed at one time
        prefetch: 2
        # Manual answer
        acknowledge-mode: manual

QOS prefetch (set unacknowledged message buffer size)

introduce

This is a kind of protection mechanism of RabbitMQ. It can prevent a large number of messages from entering consumers and causing consumer downtime when messages surge.

This value defines the maximum number of unacknowledged messages allowed on the channel to prevent too many unacknowledged messages in the unacknowledged message buffer.

Once the number reaches the configured number, RabbitMQ will stop delivering more messages on the channel.

Unless at least one unprocessed message is acknowledged, for example, assuming that there are unacknowledged messages 5, 6, 7 and 8 on the channel and the prefetch count of the channel is set to 4, RabbitMQ will not deliver any messages on the channel unless at least one unacknowledged message is acked. For example, if tag=6 is just acknowledged, RabbitMQ will perceive this message The situation arrives and another message is sent.

code implementation

This can be achieved by setting the number of message allocations.

listener:
  simple:
    # Minimum concurrent number of consumers
    concurrency: 1
    # Maximum concurrent number of consumers
    max-concurrency: 5
    # Number of messages processed at one time
    prefetch: 2
    # Manual answer
    acknowledge-mode: manual

Buffer size

min = concurrency * prefetch * number of nodes
Max = max concurrency * prefetch * number of nodes

When unacketed_msg_count < min, the queue will not be blocked. However, unacketed messages need to be processed in time. - unacketed_msg_count > = min may be blocked. - unacketed_msg_count > = max queue must be blocked.

Dead letter queue

RabbitMQ's dead letter queue is not the same as RocketMQ. It requires us to set up a switch and bind the queue. We use it semantically as a queue for storing messages that cannot be consumed.

RabbitMQ's dead letter is set by setting the dead letter parameter for the normal queue. When there are messages that cannot be consumed in the queue, these messages will be transferred to the set dead letter queue.

Cause of bad letter message

Message TTL expired
The queue has reached the maximum length (the queue is full and can no longer add data to mq)
The message is rejected (basic.reject or basic.nack) and request = false

TTL in RabbitMQ

TTL is the attribute of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in the queue, in milliseconds.

In other words, if a message has the TTL attribute set or enters the queue with the TTL attribute set, the message will become a "dead letter" if it is not consumed within the time set by the TTL. If both the TTL of the queue and the TTL of the message are configured, the smaller value will be used. There are two ways to set the TTL.

How to set TTL

Message setting TTL

Message msg = new Message(s.getBytes(StandardCharsets.UTF_8));
//Parameter 4: MessagePostProcessor: used to add / modify headers or attributes after message conversion. 
//It can also be used to modify inbound messages when the listener container and AmqpTemplate receive messages.
rabbitTemplate.convertAndSend("MqSendService-One","One",msg,correlationData->{
    correlationData.getMessageProperties().setExpiration("1000");
    return correlationData;
});


//You can also specify when creating a message
 msg.getMessageProperties().setExpiration("1000");

Queue settings TTL

@Bean
public DirectExchange directExchange(){
    Map<String, Object> args = new HashMap<>(3);
    //Declare the TTL of the queue
    args.put("x-message-ttl", 10000);
    //Parameter introduction
    //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
    return new DirectExchange("MqSendService-One",false,false,args);
}


@Bean
public Queue directQueue(){
    //The required attributes can be added continuously by the builder
    Queue queue = QueueBuilder.noDurable("TTL_Queue").ttl(100).build();
    return queue;
}

The difference between the two

If the TTL attribute of the queue is set, once the message expires, it will be discarded by the queue (if the dead letter queue is configured, it will be discarded into the dead letter queue),

The message is set to TTL mode. Even if the message expires, it will not be discarded immediately, because RabbitMQ will only check whether the first message expires. If it expires, it will be thrown into the dead letter queue. If the delay time of the first message is long and the delay time of the second message is short, the second message will not be executed first.

In addition, it should be noted that if TTL is not set, the message will never expire. If TTL is set to 0, the message will be discarded unless it can be delivered directly to the consumer at this time.

code implementation

1. Semantic declaration dead letter switch

@Bean
public DirectExchange deadExchange(){
    //Parameter introduction
    //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
    return new DirectExchange("Dead_Exchange",false,false,null);
}

2. Declare the dead letter queue and establish the binding relationship

@Bean
public Queue directQueue(){
    //Parameter introduction
    //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
    return new Queue("Dead_Queue",false,false,false,null);
}

3. Set dead letter parameters for normal queue (key)

@Bean
public Queue directQueue(){
    Map<String, Object> args = new HashMap<>(3);
    //Declare the dead letter switch bound by the current queue
    args.put("x-dead-letter-exchange", "dead_exchange");
    //Declare the dead letter routing key of the current queue
    args.put("x-dead-letter-routing-key", "dead");
    //Parameter introduction
    //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
    return new Queue("directQueue-One",false,false,false,args);
}

@Bean
public Queue directQueue2(){
    Queue queue = QueueBuilder
        .durable("dis")
        .autoDelete()
        .ttl(100)
        .deadLetterExchange("Dead_Exchange")		//Set parameters of dead letter switch
        .deadLetterRoutingKey("Dead")		//Set the routing key of dead letter queue
        .build();
    return queue;
}

Delay queue

Using dead letter queue to achieve

The delay queue of RabbitMQ can be achieved by setting the time of TTL and setting the parameters of dead letter queue.

Example: create a queue and set the TTL time, but no one listens for consumption. When the TTL time reaches, the message will enter the dead letter queue. At this time, set a consumer listening to the dead letter queue to delay consumption.

Using the official website to delay the queue plug-in to achieve

Priority queue

introduce

RabbitMQ supports setting priority for queues, so that messages in queues with high priority are consumed first.

Implementation code

@Bean
public Queue directQueue2() {
    //Set queue priority
    //args.put("x-max-priority",5)
    
    Queue queue = QueueBuilder
        //Persist and set queue name
        .durable("dis")
        //Turn on the queue priority and set the priority level
        .maxPriority(5)
        .build();
    return queue;
}

Inert queue

introduce

By default, when the producer sends messages to RabbitMQ, the messages in the queue will be stored in memory as much as possible, so that messages can be sent to consumers more quickly. Even for persistent messages, a backup will reside in memory while being written to disk.

Inert queue will store messages to disk as much as possible, and they will be loaded into memory only when consumers consume the corresponding messages. An important design goal of inert queue is to support more message storage. When consumers cannot consume messages for a long time due to various reasons (such as offline, downtime, or shutdown due to maintenance, etc.), it is necessary to have an inert queue.

code implementation

There are two modes for queues: default and lazy. Lazy is the lazy queue mode.

@Bean
public Queue directQueue2() {
    //Set inert queue
    //args.put("x-queue-mode", "lazy");
    
    Queue queue = QueueBuilder
        //Persist and set queue name
        .durable("dis")
        //Set as inert queue
        .lazy()
        .build();
    return queue;
}

Disaster protection

Message acknowledgement

introduce

Publisher message confirmation (publish confirmation)

Confirmation mode

NONE

Disable publish confirmation mode, which is the default
CORRELATED

The callback method will be triggered after the message is successfully published to the exchange
SIMPLE

After testing, there are two effects. One effect is the same as the corelated value, which will trigger the callback method;

Second, after publishing the message successfully, use rabbitTemplate to call waitForConfirms or waitForConfirmsOrDie methods, wait for the broker node to return the sending result, and determine the logic of the next step according to the returned result. Note that if the waitForConfirmsOrDie method returns false, the channel will be closed, and then the message cannot be sent to the broker.

quick get start

1. Set the release confirmation method for the configuration file

spring:
  application:
    name: produer-mq-7001
  rabbitmq:
    addresses: 127.0.0.1
    username: guest
    password: guest
    # Publishing confirmation method: NONE by default
    publisher-confirm-type: correlated

2. Configure RabbitTemplate

The callback needs to be set for publishing confirmation, but Spring is a singleton by default. If the RabbitTemplate is injected directly, it will be considered to reset the callback method when setting the publishing confirmation callback; A RabbitTemplate can only have one initial release confirmation callback.

public class RabbitTemplate extends RabbitAccessor implements ... {
    
    ...
        
    public void setConfirmCallback(ConfirmCallback confirmCallback) {
        Assert.state(this.confirmCallback == null || this.confirmCallback.equals(confirmCallback),
                     "Only one ConfirmCallback is supported by each RabbitTemplate");
        this.confirmCallback = confirmCallback;
    }
    
    ...
}

public abstract class Assert {
    public Assert() {
    }

    public static void state(boolean expression, String message) {
        if (!expression) {
            throw new IllegalStateException(message);
        }
    }
    
    ...
}

Solution:

Using multiple cases, you can achieve different message publishing and use different confirmation callbacks (against single cases)

@Bean
@Scope("prototype")
public RabbitTemplate getRabbitTemplate(ConnectionFactory connectionFactory){
    RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
    return rabbitTemplate;
}

Using the singleton, the confirmation callback is configured initially (there can only be one confirmation callback)

@Bean
public RabbitTemplate getRabbitTemplate(ConnectionFactory connectionFactory){
    RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
    rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
        @Override
        public void confirm(CorrelationData correlationData, boolean b, String s) {
            if (!b){
                ReturnedMessage dataReturned = correlationData.getReturned();
                String str = new String(dataReturned.getMessage().getBody());
                System.out.println(str);
                log.error("Message sending failed, please try again");
                return;
            }
        }
    });
    return rabbitTemplate;
}

@Autowired
private RabbitTemplate rabbitTemplate;

//The callback object of rabbitTemplate is set after dependency injection
@PostConstruct
public void init(){
    rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
        @Override
        public void confirm(CorrelationData correlationData, boolean b, String s) {
            if (!b){
                ReturnedMessage dataReturned = correlationData.getReturned();
                String str = new String(dataReturned.getMessage().getBody());
                System.out.println(str);
                log.error("Message sending failed, please try again");
                return;
            }
        }
    });
}

Fallback message

When only the producer confirmation mechanism is enabled, the switch will directly send a confirmation message to the message producer after receiving the message. If it is found that the message is not routable, the message will be directly discarded. At this time, the producer does not know the event that the message is discarded.

At this time, by setting the mandatory parameter, the message can be returned to the producer when the destination cannot be reached during message delivery, which needs to be combined with returncallback

@Bean
public RabbitTemplate getRabbitTemplate(ConnectionFactory connectionFactory){
    RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
    //true: when the switch cannot route the message, it will return the message to the producer
    //False: if it is found that the message cannot be routed, it will be discarded directly; Default false
    rabbitTemplate.setMandatory(true);
    //Set who will handle the fallback message
    rabbitTemplate.setReturnsCallback(new RabbitTemplate.ReturnsCallback() {
        @Override
        public void returnedMessage(ReturnedMessage returned) {
            System.out.println("--------Unable to route, fallback processing--------");
        }
    });
    //Set confirmation callback
    rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
        @Override
        public void confirm(CorrelationData correlationData, boolean b, String s) {
            if (!b){
                ReturnedMessage dataReturned = correlationData.getReturned();
                String str = new String(dataReturned.getMessage().getBody());
                System.out.println(str);
                log.error("Message sending failed, please try again");
                return;
            }
        }
    });
    return rabbitTemplate;
}

Consumer message confirmation

Message side acknowledgement mode

**NONE: * * do not confirm, that is, the listener directly confirms after listening to the message
**MANUAL: * * MANUAL confirmation. The consumer needs to reply manually for confirmation
**AUTO: * * the container will issue ack/nack according to whether the listener returns normally or throws an exception. Note that it is different from NONE

Spring's default request rejected configuration is true, so the consumption message will be re queued after an exception occurs.

Moreover, if there is a consumer cluster, the Nack message of a consumer will be handed over to other consumers.

Implementation of message confirmation

Method 1: configuration file

spring:
  application:
    name: consumer-mq-7100
  rabbitmq:
    addresses: 127.0.0.1
    cache:
      channel:
        size: 25
# Specify the consumer message confirmation method
    listener:
      simple:
        acknowledge-mode: manual

Method 2: @ RabbitListener specifies

@RabbitListener(bindings = @QueueBinding(
    value = @Queue(value = "directQueue-One", durable = "false"),
    exchange = @Exchange(value = "MqSendService-One", type = "direct", durable = "false"),
    key = "One"),
    ackMode = "MANUAL")			//Specify the consumer message confirmation method
public void tsAckDirectMsg(Message data, Channel channel) throws IOException {
    String str = new String(data.getBody());
    System.out.println(str + "-----: " + seq);
    System.out.println();
    seq.incrementAndGet();
    System.out.println(data.getMessageProperties().getDeliveryTag());
    System.out.println(channel.getChannelNumber());
    channel.basicAck(data.getMessageProperties().getDeliveryTag(),false);
}

channel.basicAck() method

Parameters:

long deliveryTag:

The index of the message. It is usually set to data.getMessageProperties().getDeliveryTag().

Each message has a unique deliveryTag in a channel. Each time one message is sent, the deliveryTag will be + 1, counting from 0;
The incoming deliveryTag of the confirmation message must be consistent with that in the channel, otherwise it cannot be confirmed, and the message will be set to ready status.

**Note: * * when deliveryTag is fixed with a number m, when m > deliveryTag, it will listen to consumption again through another channel.

Unconfirmed messages (deliveryTag mismatch, channel closed, connection closed or TCP connection lost) will be re queued and set to ready status. If there are other consumers, the message will be sent to other consumers. Otherwise, try repeatedly to save only consumers. Messages that are not acknowledged are set to unacknowledged.
boolean multiple:

Batch confirmation.

When set to true, messages with deliveryTag less than the passed deliveryTag parameter will be confirmed in batch.

channel.basicNack() method

The parameter has one more Boolean request. Whether to rejoin the queue or not. The first two parameters are the same as above.

Message persistence

By default, RabbitMQ ignores queues and messages when it exits or crashes for some reason.

Queue persistence

Set persistence to true when declaring the queue.

It should be noted that if the previously declared queue is not persistent, you need to delete the original queue or re create a persistent queue, otherwise an error will occur.

@Bean
public Queue directQueue(){
    //Parameter introduction
    //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
    return new Queue("directQueue-One",true,false,false,null);
}

Switch persistence

ditto.

Standby switch

preface

With the message fallback function, we can sense the delivery of messages, but for these messages that cannot be routed, we may only have a recording function, and then process them manually; And message fallback will increase the complexity of producers; So now how do you want to achieve without increasing the complexity of producers and ensuring that messages are not lost? Because the message is unreachable, it obviously cannot be realized through the dead letter queue mechanism. Therefore, it can be realized through the mechanism of this standby switch.

Implementation principle

It sets a standby switch for the switch when declaring the switch; When the primary switch receives a message that is unreachable, it forwards the message to the standby switch. It sends these messages to its bound queue. Generally, the type of the standby switch is set to Fanout (broadcast type). In this way, we can uniformly set up a consumer to listen to the queue under the switch for unified processing.

Implementation code

When the mandatory parameter and the backup switch can be used together, if both are enabled at the same time, who has the highest priority? The backup switch has the highest priority after testing

@Configuration
public class RabbitDirectConfig {
    @Bean
    public Queue alternateQueue(){
        //Parameter introduction
        //1. Queue name 2. Persistent 3. Exclusive 4. Auto delete 5. Other parameters
        Queue queue = QueueBuilder.durable("alternateQueue")
            .autoDelete()
            .build();
        return queue;
    }

    @Bean
    public FanoutExchange alternateExchange(){
        return new FanoutExchange("Alternate_Exchange",true,false,null);
    }

    @Bean
    public DirectExchange directExchange(){
        //        ExchangeBuilder exchange = ExchangeBuilder.directExchange("MqSendService-One")
        //                .durable(false)
        //                .autoDelete()
        //                .withArgument("alternate-exchange", "Alternate_Exchange");
        //Parameter introduction
        //1. Switch name 2. Persist 3. Delete automatically 4. Other parameters
        Map<String,Object> args = new HashMap<>(3);
        args.put("alternate-exchange","Alternate_Exchange");
        return new DirectExchange("MqSendService-One",false,false,args);
    }

    @Bean
    public Binding bingAlternateExchange(){
        return BindingBuilder.bind(alternateQueue())   //Bind queue
            .to(alternateExchange());      //Which switch is the queue bound to
    }

    @Bean
    public Binding bingExchange(){
        return BindingBuilder.bind(directQueue())   //Bind queue
            .to(directExchange())       //Which switch is the queue bound to
            .with("One");        //Routing key, must be specified
    }
}

Posted by sanam on Tue, 30 Nov 2021 06:39:06 -0800

Programmer Group

Springboot integration RabbitMQ details

RabbitMQ

Characteristics of RabbitMQ

AMQP

AMQP Model

Message confirmation

AMQP is a programmable protocol

RabbitMQ installation

Windows10 installation

step

Spring integrates AMQP

Official Chinese documents

GitHup translation document

Main object classes and functions of Spring AMQP

Main parameters of Spring AMQP

Spring boot integrates AMQP

Comments related to consumer monitoring

@RabbitListener

matters needing attention

@RabbitHandler

@Payload

@Header，@Headers

quick get start

RabbitMQ structure introduction

member

Virtual host

vhost feature

vhost operation

Switch Type

Direct type (default, match send)

Topic type (extended matching sending)

Fanout type (broadcast transmission)

Headers (key value pairs match, not commonly used)

Message

Message serialization

Queue

Builder creation

Construction method new

Characteristic function

Prefetch count (message allocation)

QOS prefetch (set unacknowledged message buffer size)

introduce

code implementation

Buffer size

Dead letter queue

Cause of bad letter message

TTL in RabbitMQ

How to set TTL

Message setting TTL

Queue settings TTL

The difference between the two

code implementation

Delay queue

Using dead letter queue to achieve

Using the official website to delay the queue plug-in to achieve

Priority queue

introduce

Implementation code

Inert queue

introduce

code implementation

Disaster protection

Message acknowledgement

introduce

Publisher message confirmation (publish confirmation)

Confirmation mode

quick get start

Fallback message

Consumer message confirmation

Message side acknowledgement mode

Implementation of message confirmation

channel.basicAck() method

channel.basicNack() method

Message persistence

Queue persistence

Switch persistence

Standby switch

preface

Implementation principle