1, Message queue

Basic requirements of message queuing

• Message order preservation: the generation and consumption of messages are in order, and the message queue needs to ensure the order of consumption.
• Duplicate message processing: the producer may send duplicate messages, and the idempotency of consumption shall be guaranteed.
• Message reliability assurance: messages cannot be lost. After consumption fails, it is necessary to ensure that there is a mechanism for re consumption.

Guarantee of message queue reliability

• Producer lost data
• Consumer lost data
• The message queue itself loses data

2, list based message queue

Redis's sub/pub mechanism can implement some message queuing mechanisms, but in case of network connection interruption and database downtime, data is easy to be lost and cannot be consumed repeatedly. The list based consumption queue can simply realize some functions of the consumption queue.

Implementation instruction

• LPUSH: insert a message into the queue;
• RPOP: queue consumes a message;
• BRPOP: blocking consumption of a message;
• BRPOPLPUSH: plug a message into another consumption queue while consuming it in a blocking manner (for repeated consumption in case of consumption failure);

Use example

# Insert a data with message sequence 00001 and message value msg:5 into the consumption queue mq
127.0.0.1:6379> lpush mq 00001:msg:5
(integer) 1
# Consume a message
127.0.0.1:6379> rpop mq 
"00001:msg:5"

127.0.0.1:6379> lpush mq 00002:msg:7 00003:msg:4
(integer) 2
# Blocking (3s timeout) consumption of a message
127.0.0.1:6379> brpop mq 3
1) "mq"
2) "00002:msg:7"
(2.51s)
127.0.0.1:6379> brpop mq 3
1) "mq"
2) "00003:msg:4"

127.0.0.1:6379> lpush mq 00004:msg:9
(integer) 1
# Blocking (3s timeout) consumes a message and puts the consumed data into the mq2 queue to prevent data loss when consumption fails.
127.0.0.1:6379> brpoplpush mq mq2 3
"00003:msg:9"
# Delete the message in mq2 after the consumption is successful
127.0.0.1:6379> rpop mq2
"00003:msg:9"

Existing problems

1. The business needs to ensure that the sequence ID is unique to avoid repeated consumption.
2. BRPOPLPUSH in order to prevent re consumption after consumption failure, an additional consumption queue will be generated and occupy memory.
3. The concept of consumption group is not supported. When the consumption speed is slow, it will lead to message queue data backlog.
4. The message queue itself is based on redis, and there is a risk of data loss.

3, Message queue based on streams

After Redis5.0, streams data structure is newly supported to support fully functional message queue, new consumption group consumption, ACK confirmation and other complete mechanisms.

Implementation instruction

• XADD: insert messages to ensure order and automatically generate globally unique ID s;
• XREAD: used to read messages and read data by ID;
• XREADGROUP: read messages in the form of consumption group;
• XPENDING: used to query messages that have been read but not confirmed by all consumers in each consumption group;
• XACK: used to confirm to the message queue that the message processing has been completed;

Use example

• xadd key ID field string [field string ...]
  • ID: you can specify a unique serial number by yourself. Generally, * is used to specify that the system will automatically generate an ID serial number: [sequence starting with millisecond timestamp - 0], such as "1633939083516-0"

127.0.0.1:6379> xadd mq * money -5
"1633939083516-0"
127.0.0.1:6379> xadd mq * money -5 money +10
"1633953837863-0"

• xrange key start end [COUNT count]

127.0.0.1:6379> xrange mq - +
1) 1) "1633939083516-0"
   2) 1) "money"
      2) "-5"
2) 1) "1633953837863-0"
   2) 1) "money"
      2) "-5"
      3) "money"
      4) "+10"

• xread [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]
  • count: number of reads
  • block: the number of blocking milliseconds. If it is not set, it represents non blocking reading
  • key: message queue name
  • id: message sequence number. 0-0 represents the minimum start and $represents the start from now on.

127.0.0.1:6379> xread count 2 block 5000 streams mq 0-0
1) 1) "mq"
   2) 1) 1) "1633939083516-0"
         2) 1) "money"
            2) "-5"
      2) 1) "1633953837863-0"
         2) 1) "money"
            2) "-5"
            3) "money"
            4) "+10"

• xgroup [CREATE key groupname id-or-$ ] [SETID key id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername]

# Create a group1 group of consumption mq queue, and start consumption from ID 0-0.
127.0.0.1:6379> xgroup create mq group1 0-0
OK

• xreadgroup GROUP group consumer [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]

# consumer1 of group1 group starts to consume mq queue from 0-0
127.0.0.1:6379> xreadgroup GROUP group1 consumer1 STREAMS mq 0-0
1) 1) "mq"
   2) 1) 1) "1633939083516-0"
         2) 1) "money"
            2) "-5"
      2) 1) "1633953837863-0"
         2) 1) "money"
            2) "-5"
            3) "money"
            4) "+10"

• xpending key group [start end count] [consumer]

# Query the message currently read but confirmed consumption
127.0.0.1:6379> xpending mq group1 - + 2 consumer1
1) 1) "1633939083516-0"
   2) "consumer1"
   3) (integer) 672137
   4) (integer) 1
2) 1) "1633953837863-0"
   2) "consumer1"
   3) (integer) 672137
   4) (integer) 1

• xack key group ID [ID ...]

# Confirm 1633939083516-0 message
127.0.0.1:6379> xack mq group1 1633939083516-0
(integer) 1
127.0.0.1:6379> xpending mq group1 - + 2 consumer1
1) 1) "1633953837863-0"
   2) "consumer1"
   3) (integer) 840477
   4) (integer) 1

Existing problems

1. Redis5.0 and later versions are supported.
2. The message queue itself is based on redis, and there is still a risk of data loss.
3. Lightweight message queue, suitable for data loss insensitive services (SMS, notification).