About rabbitmq prefetch count
1-1: cause
I've always had a vague impression of prefetch count. I only know that instead of the original broker polling for consumer s, I can pull down n items at a time, but when I send message s one by one through the interface, I still poll, so I have this verification blog.
1-2: experimental code
- Configuration class
package com.example.springbootrabbitmq.configuration;
import com.example.springbootrabbitmq.MyConfirmCallback;
import com.example.springbootrabbitmq.MyReturnCallback;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.AcknowledgeMode;
import org.springframework.amqp.rabbit.config.SimpleRabbitListenerContainerFactory;
import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.amqp.SimpleRabbitListenerContainerFactoryConfigurer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* @author xx
* @date 2021/10/5 16:24
*/
@Configuration
@Slf4j
public class RabbitmqConfig {
@Autowired
private CachingConnectionFactory connectionFactory;
//Automatically assemble the container factory configuration class instance where the message listener is located
@Autowired
private SimpleRabbitListenerContainerFactoryConfigurer factoryConfigurer;
// @Bean
// public MessageConverter jsonMessageConverter() {
// return new Jackson2JsonMessageConverter();
// }
@Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
// rabbitTemplate.setConfirmCallback(new MyConfirmCallback());
// rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
// if (ack) {
log.info("Sent successfully");
// } else {
correlationData.getReturned().getMessage().getMessageProperties().getMessageId();
log.info("fail in send");
// }
// });
// rabbitTemplate.setReturnsCallback(new MyReturnCallback());
return rabbitTemplate;
}
/**
* Different container configurations can be made for different consumers to realize different configurations for multiple consumer applications.
*/
@Bean(name = "singleListenerContainer")
public SimpleRabbitListenerContainerFactory listenerContainer() {
//Defines the container factory where the message listener resides
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
//Sets the instance used by the container factory
factory.setConnectionFactory(connectionFactory);
//Set the format of the message in transmission. Here, the message is transmitted in JSON format
factory.setMessageConverter(new Jackson2JsonMessageConverter());
// //Sets the initial number of concurrent consumer instances. Here for 1
// factory.setConcurrentConsumers(1);
// //Sets the maximum number of concurrent consumer instances. Here for 1
// factory.setMaxConcurrentConsumers(1);
// //Set the number of messages pulled by each instance in the concurrent consumer instance - 1 here
// factory.setPrefetchCount(1);
// Turn off auto answer
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
// Set unfair distribution, change to read one message at a time, and do not deliver the next message before the consumer does not acknowledge the receipt, instead of the default polling;
// In other words, after I finish processing, I will accept the next delivery, which belongs to the control of the consumer
// If it is not set, the polling method is used to listen to the queue, one by one
factory.setPrefetchCount(3);
return factory;
}
}
- Queue switch configuration
//Queue name: springDirectQueue
@Bean
public Queue springDirectQueue() {
// durable: whether to persist. The default is false. Persistent queue: it will be stored on the disk, otherwise the queue will disappear when your computer is turned off.
// exclusive: it is also false by default. It can only be used by the currently created connection, and the queue will be deleted after the connection is closed. This reference takes precedence over durable
// autoDelete: whether to delete automatically. When no producer or consumer uses this queue, the queue will be deleted automatically.
//Generally, it's good to set the persistence of the queue. The other two are false by default
return new Queue("springDirectQueue", true);
}
@Bean
public Queue springDirectQueue2() {
return new Queue("springDirectQueue2", true);
}
//Name of Direct switch: springDirectExchange
@Bean
public DirectExchange TestDirectExchange() {
return new DirectExchange("springDirectExchange", true, false);
}
//Use two queues to bind the queue to the switch, and set the matching keys: springdirectrouting1 and springdirectrouting2 to experiment with the role of routingKey
@Bean
public Binding bindingDirectQueue1() {
return BindingBuilder.bind(springDirectQueue()).to(TestDirectExchange()).with("springDirectRouting1");
}
@Bean
public Binding bindingDirectQueue2() {
return BindingBuilder.bind(springDirectQueue2()).to(TestDirectExchange()).with("springDirectRouting2");
}
- Producer.java
package com.example.springbootrabbitmq.controller;
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessageBuilder;
import org.springframework.amqp.core.MessageProperties;
import org.springframework.amqp.rabbit.connection.CorrelationData;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import javax.annotation.Resource;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.logging.Logger;
/**
* @author xx
* @date 2021/10/5 16:10
*/
@RestController
@RequestMapping("/directProducer")
@Slf4j
public class DirectProducerController {
@Resource
private RabbitTemplate rabbitTemplate;
@GetMapping("/sendMessage")
public void sendMessage(String msg) {
Map<String, String> msgMap = new HashMap<>();
msgMap.put("message", msg);
String messageJson = JSONObject.toJSONString(msgMap);
Message message = MessageBuilder
.withBody(messageJson.getBytes())
.setContentType(MessageProperties.CONTENT_TYPE_JSON)
.setContentEncoding("utf-8")
.setMessageId(UUID.randomUUID() + "")
.build();
log.info("Producer send:" + new String(message.getBody(), StandardCharsets.UTF_8));
CorrelationData correlationData = new CorrelationData();
correlationData.setId("1");
rabbitTemplate.convertAndSend("springDirectExchange", "springDirectRouting1", message, correlationData);
}
}
- consumer.java
/*******************To verify the messages in the same queue, the polling method start is adopted by default********************/
@RabbitListener(queues = "springDirectQueue", containerFactory = "singleListenerContainer")
public void processMessageQueue(Message message, Channel channel) throws Exception {
String messageString = new String(message.getBody(), StandardCharsets.UTF_8);
log.info("--------springDirectQueue_consumer1 Accepted:{} --------", messageString);
int channelNumber = channel.getChannelNumber();
log.info("--------springDirectQueue_consumer1-channelNumber Is:{}", channelNumber);
long deliveryTag = message.getMessageProperties().getDeliveryTag();
log.info("--------springDirectQueue_consumer1-deliveryTag: " + deliveryTag);
TimeUnit.SECONDS.sleep(3);
channel.basicAck(deliveryTag, false);
// channel.basicNack(deliveryTag, false, true);
log.info("--------springDirectQueue_consumer1 End of processing--------");
}
@RabbitListener(queues = "springDirectQueue", containerFactory = "singleListenerContainer")
public void processMessageQueue2(Message message, Channel channel) throws Exception {
String messageString = new String(message.getBody(), StandardCharsets.UTF_8);
log.info("--------springDirectQueue_consumer2 Accepted:{}--------", messageString);
int channelNumber = channel.getChannelNumber();
log.info("--------springDirectQueue_consumer2-channelNumber Is:{}", channelNumber);
long deliveryTag = message.getMessageProperties().getDeliveryTag();
log.info("--------springDirectQueue_consumer2-deliveryTag: " + deliveryTag);
TimeUnit.SECONDS.sleep(3);
channel.basicAck(deliveryTag, false);
log.info("--------springDirectQueue_consumer2 End of processing--------");
}
1-3: test process
First, comment out the two conusmer s, send 10 messages to the queue through the interface, and give your own message number. Of course, you can cycle 10 times. This is the best. I ordered it here.. Through the rabbitmq web management page, you can see that there are 10 messages in the queue. At this time, start the project and let two consumers listen to the queue. Through the grep console plug-in, we can view the log information of two consumers. As follows:
-
conusmer1
2021-10-24 16:42:47.198 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 accepts: {"message": "test directExchange1"}-------- 2021-10-24 16:42:47.199 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer1 channelnumber: 12 2021-10-24 16:42:47.199 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1-deliveryTag: 1 2021-10-24 16:42:50.204 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 end of processing-------- 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 accepts: {"message": "testing directexchange 2"}-------- 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer1 channelnumber: 12 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1-deliveryTag: 2 2021-10-24 16:42:53.208 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 end of processing-------- 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 accepts: {"message": "test directExchange3"}-------- 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer1 channelnumber: 12 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1-deliveryTag: 3 2021-10-24 16:42:56.212 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 end of processing-------- 2021-10-24 16:42:56.213 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 accepts: {"message": "test directExchange7"}-------- 2021-10-24 16:42:56.214 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer1 channelnumber: 12 2021-10-24 16:42:56.214 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1-deliveryTag: 4 2021-10-24 16:42:59.214 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 end of processing-------- 2021-10-24 16:42:59.215 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 accepts: {"message": "test directExchange10"}-------- 2021-10-24 16:42:59.215 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer1 channelnumber: 12 2021-10-24 16:42:59.216 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1-deliveryTag: 5 2021-10-24 16:43:02.216 INFO 30448 --- [ntContainer#7-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer1 end of processing-------- -
consumer2
2021-10-24 16:42:47.198 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 accepts: {"message": "test directExchange4"}-------- 2021-10-24 16:42:47.199 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer2 channelnumber: 13 2021-10-24 16:42:47.199 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2-deliveryTag: 1 2021-10-24 16:42:50.204 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 end of processing-------- 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 accepts: {"message": "test directExchange5"}-------- 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer2 channelnumber: 13 2021-10-24 16:42:50.205 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2-deliveryTag: 2 2021-10-24 16:42:53.208 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 end of processing-------- 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 accepts: {"message": "test directExchange6"}-------- 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer2 channelnumber: 13 2021-10-24 16:42:53.209 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2-deliveryTag: 3 2021-10-24 16:42:56.213 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 end of processing-------- 2021-10-24 16:42:56.213 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 accepts: {"message": "test directExchange8"}-------- 2021-10-24 16:42:56.214 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer2 channelnumber: 13 2021-10-24 16:42:56.214 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2-deliveryTag: 4 2021-10-24 16:42:59.214 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 end of processing-------- 2021-10-24 16:42:59.215 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 accepts: {"message": "test directExchange9"}-------- 2021-10-24 16:42:59.215 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_ Consumer2 channelnumber: 13 2021-10-24 16:42:59.215 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2-deliveryTag: 5 2021-10-24 16:43:02.216 INFO 30448 --- [ntContainer#8-1] c.e.s.c.DirectConsumerController : --------springDirectQueue_consumer2 end of processing--------
1-4: draw conclusions
The message numbers received by consumer1 are 1, 2, 3, 7, 10; The message numbers received by consumer2 are 4, 5, 6, 8, 9. Thus, it can be concluded that at the beginning of listening, consumer1 and consumer2 pull three messages at the same time (the same as our configuration), but after that, each person does not pull three messages. The broker will deliver them if the number of unack nowledges of each consumer does not reach the value of prefetch count. The author's summary feels very in place. Let's quote.
prefetch allows you to specify the maximum number of unacknowledged messages per consumer. In short, it is used to specify how many messages a consumer can get from the Rabbit at a time and store them in the client (the client library of various languages provided by RabbitMQ). Once the buffer is full, Rabbit will stop posting new messages to the consumer until it sends an ack.
Suppose the prefetch value is set to 10 and there are two consumers. This means that each consumer will prefetch 10 messages from the queue to the local cache for consumption. At the same time, the unacketed number of the channel becomes 20. The order of rabbit delivery is to deliver 10 messages to consumer1 first, and then 10 messages to consumer2. If a new message needs to be delivered at this time, first judge whether the unacknowledged number of the channel is equal to 20. If so, the message will not be delivered to the consumer, and the message will continue to stay in the queue. After that, the consumer acks a message. Unackaged is equal to 19. Rabbit determines which consumer has unackaged less than 10 and delivers it to which consumer.
Author: zcliu
Link: https://www.jianshu.com/p/4d043d3045ca
Source: Jianshu
In this way, even after the prefetch count is set, the consumer knows how to pull messages.