Notes on the rabbitmq dead letter queue used in development

Application scenarios for delayed queues

1. Cancellation of unpaid orders on time
 2. Timely cleaning of cached objects, idle connections, etc.
3. Within 30 minutes after successful placing of the order, notices should be sent at different time intervals (1 min, 3 min, 10 min once).

1. Setting queue expiration time

$this->channel->queue_declare(
            $this->retry_queue(),
            false,
            true,
            false,
            false,
            false,
            new AMQPTable(
                [
                    # Without setting x-dead-letter-routing-key, the original routing_key is used and automatically returns to the original queue after 10s expiration. Then the x-dead-letter-exchange switch needs to bind the original queue.
                    'x-dead-letter-exchange' => $this->retry_exchange(),
                    
                    # 10s
                    'x-message-ttl' => 10000,
                ]
            )
        );

All messages pushed to the queue (without ttl) will expire after 10 seconds, according to the original routing_key, enter the specified exchange, and then enter the specified queue.

2. Setting the expiration time of messages

$message = new AMQPMessage(
    'msg',
    array(
        # Message persistence
        'delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSITENT,
        
        # ttl expiration time
        'expiration' => 50000,
    )
);

Each message sets the same expiration time, and the message will expire when it expires.

3. Setting the expiration time of queues and messages at the same time

If set at the same time, the expiration time of the message will depend on a smaller value, such as the queue's `x-message-ttl'set to 10s and the message's `expiration' set to 50s, then the message will expire after 10s.

4. Follow-up

The dead letter queue can work properly if the ttl of the queue is set alone or the same message expiration time is set alone. However, setting different message expiration times may prevent the normal use of dead-letter queues.

Queue without ttl

$this->channel->queue_declare(
            $this->retry_queue(),
            false,
            true,
            false,
            false,
            false,
            new AMQPTable(
                [
                    # Without setting x-dead-letter-routing-key, the original routing_key is used and automatically returns to the original queue after 10s expiration. Then the x-dead-letter-exchange switch needs to bind the original queue.
                    'x-dead-letter-exchange' => $this->retry_exchange(),
                ]
            )
        );

The first message set expires at 500s, pushing the queue first

$message = new AMQPMessage(
    'msg',
    array(
        # Message persistence
        'delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSITENT,
        
        # ttl expiration time
        'expiration' => 500000,
    )
);

The second message set expires for 5s and then advances the queue

$message = new AMQPMessage(
    'msg',
    array(
        # Message persistence
        'delivery_mode' => AMQPMessage::DELIVERY_MODE_PERSITENT,
        
        # ttl expiration time
        'expiration' => 5000,
    )
);

It was found that there were two messages in the queue after 5s. The second message does not "really expire". The reason is that the message at the head of the queue has not expired. The dead-letter queue of rabbitmq is based on the first message.

5. CONCLUSION

When MQ checks the first message in the queue and finds that it has not expired, it will not continue to check the message after. Even if the later message expires, it will not be able to flow to other queues because it is not in the head of the queue, which is determined by the characteristics of MQ queue. You can't consume the information in the middle of the queue. The queue must be first in, first out.

For the method of setting queue TTL attributes, once the message expires, it will be erased from the queue, and setting message header attributes, even if the message expires, will not be erased from the queue immediately, because each message expires when it is decided before it is about to be delivered to the consumer, why are the two methods inconsistent? Because in the first method, the expired messages in the queue must be in the head of the queue, RabbitMQ only needs to scan the expired messages regularly from the head of the queue, while in the second method, the expiration time of each message is different. If you want to delete all expired messages, it is necessary to scan the whole queue, so it is better to wait for this to disappear. Interest is about to be consumed before deciding whether it is overdue, if it is overdue, then delete it.

Official narrative

"Only when expired messages reach the head of a queue will they actually be discarded (or dead-lettered)." Only when expired messages reach the top of the queue (the head of the queue), will they really be discarded or entered into the dead-letter queue.

Reference

http://blog.lbanyan.com/rabbitmq_delay/
https://blog.csdn.net/u013256816/article/details/54916011
https://juejin.im/post/5b5e52ecf265da0f716c3203