Use of Redis Stream type

Keywords: watermark shadow entry streams

1, Background

Recently, after looking at the knowledge of redis, I found that a new data type Stream has been generated in redis5, which is somewhat similar to the design of kafka and can be used as a simple message queue.

2, Characteristics of Stream type in redis

It is persistent and can ensure no data loss.
Support message multicast and packet consumption.
Support the ordering of messages.

3, Structure of Stream

Explanation:

Consumer Group: Consumer Group, which is created by using the XGROUP CREATE command. There can be multiple consumers in a Consumer Group. These consumers are competitive.
1. The same message can only be obtained by a consumer in this consumer group.
2. Multiple consumers are independent of each other and do not interfere with each other.
Consumer: consumer consumption message.
last_delivered_id: this id ensures that a message can only be obtained by one consumer in the same consumer group. Every time a consumer in the consumer group reads this message, this last_ delivered_ The value of id will move back one bit to ensure that consumers will not read duplicate messages.
pending_ids: records the list of message IDS read by the consumer, but these messages may not have been processed. If you think a message is processed, you need to call the ack command. This ensures that a message must be executed once.
Message content: the format of a key value pair.
ID of message in Stream: by default, the ID is * and redis can automatically generate one. The format is timestamp serial number. You can also specify it yourself. Generally, the ID generated by default can be used, and the ID number generated later is larger than that generated before.

4, Stream command

1. XADD adds a message to the end of the Stream

1. Command format

xadd key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold [LIMIT count]] *|ID field value [field value ...]

2. Examples

The xadd command returns the ID of the data, XX yy (XX refers to the number of milliseconds and yy refers to the number of messages in this millisecond)

1. Add a piece of data to the stream,

127.0.0.1:6379&gt; xadd stream-key * username zhangsan # Add a data * whose username is zhangsan to the stream key, indicating that the id is automatically generated
"1635999858912-0" # The ID is returned
127.0.0.1:6379&gt; keys *
1) "stream-key" # You can see that the stream is created automatically
127.0.0.1:6379&gt;

2. Adding data to a stream does not automatically create a stream

127.0.0.1:6379&gt; xadd not-exists-stream nomkstream * username lisi # The join failed because the nomkstream parameter was specified and not exists stream did not exist before
(nil)
127.0.0.1:6379&gt; keys *
(empty array)
127.0.0.1:6379&gt;

3. Manually specify the value of the ID

127.0.0.1:6379&gt; xadd stream-key 1-1 username lisi # Here, the value of id is 1-1 passed by itself, rather than generated automatically using *
"1-1" # The value of id is returned
127.0.0.1:6379&gt;

4. Set a fixed size Stream

1. Specify the exact size of the Stream

Specifying the size of the specified Stream consumes slightly more performance than specifying the size of the Stream vaguely.

2. Specifies the size of the Stream

127.0.0.1:6379&gt; xadd stream-key maxlen ~ 1 * first first
"1636001034141-0"
127.0.0.1:6379&gt; xadd stream-key maxlen ~ 1 * second second
"1636001044506-0"
127.0.0.1:6379&gt; xadd stream-key maxlen ~ 1 * third third
"1636001057846-0"
127.0.0.1:6379&gt; xinfo stream stream-key
 1) "length"
 2) (integer) 3
 3) "radix-tree-keys"
 4) (integer) 1
 5) "radix-tree-nodes"
 6) (integer) 2
 7) "last-generated-id"
 8) "1636001057846-0"
 9) "groups"
10) (integer) 0
11) "first-entry"
12) 1) "1636001034141-0"
    2) 1) "first"
       2) "first"
13) "last-entry"
14) 1) "1636001057846-0"
    2) 1) "third"
       2) "third"
127.0.0.1:6379&gt;

~Blur the size of the specified stream. You can see that the specified value is 1, which has actually reached 3

2. XRANGE viewing messages in a Stream

1. Command format

xrange key start end [COUNT count]

2. Prepare data

127.0.0.1:6379&gt; multi
OK
127.0.0.1:6379(TX)&gt; xadd stream-key * username zhangsan
QUEUED
127.0.0.1:6379(TX)&gt; xadd stream-key * username lisi
QUEUED
127.0.0.1:6379(TX)&gt; exec
1) "1636003481706-0"
2) "1636003481706-1"
127.0.0.1:6379&gt; xadd stream-key * username wangwu
"1636003499055-0"
127.0.0.1:6379&gt;

Using the redis transaction operation, multiple pieces of data generated in the same millisecond are obtained, with the same timestamp and different serial numbers

3. Examples

1. Get all data (- and + use)

127.0.0.1:6379&gt; xrange stream-key - +
1) 1) "1636003481706-0"
   2) 1) "username"
      2) "zhangsan"
2) 1) "1636003481706-1"
   2) 1) "username"
      2) "lisi"
3) 1) "1636003499055-0"
   2) 1) "username"
      2) "wangwu"
127.0.0.1:6379&gt;

-: indicates the value of the minimum id

+: indicates the value of the maximum id

2. Get the data within the specified id range and close the interval

127.0.0.1:6379&gt; xrange stream-key 1636003481706-1 1636003499055-0
1) 1) "1636003481706-1"
   2) 1) "username"
      2) "lisi"
2) 1) "1636003499055-0"
   2) 1) "username"
      2) "wangwu"
127.0.0.1:6379&gt;

3. Get the data within the specified id range and open the interval

127.0.0.1:6379&gt; xrange stream-key (1636003481706-0 (1636003499055-0
1) 1) "1636003481706-1"
   2) 1) "username"
      2) "lisi"
127.0.0.1:6379&gt;

(: indicates open interval

4. Get all data after a certain millisecond

127.0.0.1:6379&gt; xrange stream-key 1636003481706 +
1) 1) "1636003481706-0"
   2) 1) "username"
      2) "zhangsan"
2) 1) "1636003481706-1"
   2) 1) "username"
      2) "lisi"
3) 1) "1636003499055-0"
   2) 1) "username"
      2) "wangwu"
127.0.0.1:6379&gt;

Write milliseconds directly without writing the subsequent serial number.

5. Get a single piece of data

127.0.0.1:6379&gt; xrange stream-key 1636003499055-0 1636003499055-0
1) 1) "1636003499055-0"
   2) 1) "username"
      2) "wangwu"
127.0.0.1:6379&gt;

The values of start and end are the same, so you can get the single pick data.

6. Get fixed number of data

127.0.0.1:6379&gt; xrange stream-key - + count 1
1) 1) "1636003481706-0"
   2) 1) "username"
      2) "zhangsan"
127.0.0.1:6379&gt;

Limit with count

3. XREVRANGE reverses the view of messages in the Stream

XREVRANGE key end start [COUNT count]

The method of use is similar to XRANGE, slightly.

4. XDEL delete message

1. Command format

xdel key ID [ID ...]

2. Prepare data

127.0.0.1:6379&gt; xadd stream-key * username zhangsan
"1636004176924-0"
127.0.0.1:6379&gt; xadd stream-key * username lisi
"1636004183638-0"
127.0.0.1:6379&gt; xadd stream-key * username wangwu
"1636004189211-0"
127.0.0.1:6379&gt;

3. Examples

**Requirements: * * add 3 messages to the Stream, and then delete the second message

127.0.0.1:6379&gt; xdel stream-key 1636004183638-0
(integer) 1 # Returns the number of deleted records
127.0.0.1:6379&gt; xrang stream -key - +
127.0.0.1:6379&gt; xrange stream-key - +
1) 1) "1636004176924-0"
   2) 1) "username"
      2) "zhangsan"
2) 1) "1636004189211-0"
   2) 1) "username"
      2) "wangwu"
127.0.0.1:6379&gt;

be careful:

It should be noted that when we delete a message from the Stream, it is not really deleted, but marked for deletion. At this time, the message still occupies the content space. If all messages in all streams are marked for deletion, the memory space will be recovered at this time. However, the Stream will not be deleted.

6. XLEN viewing the length of elements in a Stream

1. Command format

xlen key

2. Examples

View the length of the element in the Stream

127.0.0.1:6379&gt; xadd stream-key * username zhangsan
"1636004690578-0"
127.0.0.1:6379&gt; xlen stream-key
(integer) 1
127.0.0.1:6379&gt; xlen not-exists-stream-key
(integer) 0
127.0.0.1:6379&gt;

be careful:

If the key after xlen does not exist, it returns 0; otherwise, it returns the number of elements.

7. XTRIM trims the elements in the Stream

1. Command format

xtrim key MAXLEN|MINID [=|~] threshold [LIMIT count]

2. Prepare data

127.0.0.1:6379&gt;  xadd stream-key * username zhangsan
"1636009745401-0"
127.0.0.1:6379&gt; multi
OK
127.0.0.1:6379(TX)&gt; xadd stream-key * username lisi
QUEUED
127.0.0.1:6379(TX)&gt; xadd stream-key * username wangwu
QUEUED
127.0.0.1:6379(TX)&gt; exec
1) "1636009763955-0"
2) "1636009763955-1"
127.0.0.1:6379&gt; xadd stream-key * username zhaoliu
"1636009769625-0"
127.0.0.1:6379&gt;

3. Examples

1. maxlen exact limit

127.0.0.1:6379&gt; xtrim stream-key maxlen 2 # Keep the last 2 messages
(integer) 2
127.0.0.1:6379&gt; xrange stream-key - + # You can see that the two previously added messages have been deleted
1) 1) "1636009763955-1"
   2) 1) "username"
      2) "wangwu"
2) 1) "1636009769625-0"
   2) 1) "username"
      2) "zhaoliu"
127.0.0.1:6379&gt;

The above means to keep the last two messages in the Stream, Stream key.

2. minid fuzzy limit

minid is to delete data smaller than this id, which is not tested during local testing. It is omitted.

8. XREAD independent consumption message

XREAD only reads messages and does not delete them after reading. Reading messages with XREAD is completely independent of the consumer group. Multiple clients can read messages at the same time.

1. Command format

xread [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]

2. Prepare data

127.0.0.1:6379&gt; xadd stream-key * username zhangsan
"1636011801365-0"
127.0.0.1:6379&gt; xadd stream-key * username lisi
"1636011806261-0"
127.0.0.1:6379&gt; xadd stream-key * username wangwu
"1636011810905-0"
127.0.0.1:6379&gt;

3. Examples

1. Get the data whose user name is wangwu

127.0.0.1:6379&gt; xread streams stream-key 1636011806261-0 # The id of lisi is written here, that is, the read data needs to be & gt; 1636011806261-0
1) 1) "stream-key"
   2) 1) 1) "1636011810905-0"
         2) 1) "username"
            2) "wangwu"

2. Get 2 pieces of data

127.0.0.1:6379&gt; xread count 2 streams stream-key 0-0
1) 1) "stream-key"
   2) 1) 1) "1636011801365-0"
         2) 1) "username"
            2) "zhangsan"
      2) 1) "1636011806261-0"
         2) 1) "username"
            2) "lisi"
127.0.0.1:6379&gt;

count limits the last message to be read in a single time, because there may not be so many current reads.

3. Non blocking reading of Stream tail data

That is, the next message at the end of the queue is always nil in non blocking mode

127.0.0.1:6379&gt; xread streams stream-key $
(nil)

4. Blocking the reading of Stream tail data

be careful:

$means to read the latest message in the queue, not the last message in the Stream. After the xread block is executed, the xread block will not return until the message is added with xadd again.
block 0 indicates permanent blocking, and the blocking is contacted only when the message arrives. block 1000 indicates blocking for 1000ms. If no message arrives for 1000ms, nil is returned
Xread is used for sequential consumption. When xread is used for sequential messages, you need to remember the returned message id. at the same time, when xread is called next time, you need to pass in the last returned message id.
xread reads the message and completely ignores the consumption group. At this time, the Stream can be understood as an ordinary list.

9. Consumer group related operations

1. Consumer group command

2. Prepare data

1. The name of the created Stream is Stream key

2. Create 2 messages, aa and bb

127.0.0.1:6379&gt; xadd stream-key * aa aa
"1636362619125-0"
127.0.0.1:6379&gt; xadd stream-key * bb bb
"1636362623191-0"

3. Create consumer group

1. Create a consumer group to consume from scratch

xgroup create stream-key(Stream name) g1(Consumer group name) 0-0(It means spending from scratch)

2. Create a consumer group that consumes the latest message from the Stream

xgroup create stream-key g2 $

$indicates consumption from the last element, excluding the last element in the Stream, that is, consumption of the latest message.

4. Create a consumer group to consume after a message

xgroup create stream-key g3 1636362619125-0  #1636362619125-0 this is the id value of the above aa message

1636362619125-0 the specific id of a message. Messages in this g3 consumer group are greater than & gt; This id message.

3. Read message from consumer

127.0.0.1:6379&gt; xreadgroup group g1(Consumer group name) c1(Consumer name, automatically created) count 3(Read 3) streams stream-key(Stream name) &gt;(Start reading from a message in the consumer group that has not been assigned to another consumer)
1) 1) "stream-key"
   2) 1) 1) "1636362619125-0"
         2) 1) "aa"
            2) "aa"
      2) 1) "1636362623191-0"
         2) 1) "bb"
            2) "bb"
127.0.0.1:6379&gt; xreadgroup group g2 c1 count 3 streams stream-key &gt;
(nil) # nil is returned because the g2 consumer group is read from the latest message ($) when creating the consumer group. You need to execute the 'xadd' command in another window before you can read the message again
127.0.0.1:6379&gt; xreadgroup group g3 c1 count 3 streams stream-key &gt;  #Only one message is read because the id of aa message is specified when creating the consumer group, and the id of bb message is greater than aa, so it is read.
1) 1) "stream-key"
   2) 1) 1) "1636362623191-0"
         2) 1) "bb"
            2) "bb"
127.0.0.1:6379&gt;

4. Read consumer's pending message

127.0.0.1:6379&gt; xgroup create stream-key g4 0-0
OK
127.0.0.1:6379&gt; xinfo consumers stream-key g1
1) 1) "name"
   2) "c1"
   3) "pending"
   4) (integer) 2
   5) "idle"
   6) (integer) 88792
127.0.0.1:6379&gt; xinfo consumers stream-key g4
(empty array)
127.0.0.1:6379&gt; xreadgroup group g1 c1 count 1 streams stream-key 1636362619125-0
1) 1) "stream-key"
   2) 1) 1) "1636362623191-0"
         2) 1) "bb"
            2) "bb"
127.0.0.1:6379&gt; xreadgroup group g4 c1 count 1 block 0 streams stream-key 1636362619125-0
1) 1) "stream-key"
   2) (empty array)
127.0.0.1:6379&gt;

5. Transfer consumer messages

127.0.0.1:6379&gt; xpending stream-key g1 - + 10 c1
1) 1) "1636362619125-0"
   2) "c1"
   3) (integer) 2686183
   4) (integer) 1
2) 1) "1636362623191-0"
   2) "c1"
   3) (integer) 102274
   4) (integer) 7
127.0.0.1:6379&gt; xpending stream-key g1 - + 10 c2
(empty array)
127.0.0.1:6379&gt; xclaim stream-key g1 c2 102274 1636362623191-0
1) 1) "1636362623191-0"
   2) 1) "bb"
      2) "bb"
127.0.0.1:6379&gt; xpending stream-key g1 - + 10 c2
1) 1) "1636362623191-0"
   2) "c2"
   3) (integer) 17616
   4) (integer) 8
127.0.0.1:6379&gt;

It can also be implemented through xautocliim.

6. Some monitoring commands

1. View pending messages for consumers in the consumer group

127.0.0.1:6379&gt; xpending stream-key g1 - + 10 c2
1) 1) "1636362623191-0"
   2) "c2"
   3) (integer) 1247680
   4) (integer) 8
127.0.0.1:6379&gt;

2. View consumer information in a consumer group

127.0.0.1:6379&gt; xinfo consumers stream-key g1
1) 1) "name"
   2) "c1"
   3) "pending"
   4) (integer) 1
   5) "idle"
   6) (integer) 1474864
2) 1) "name"
   2) "c2"
   3) "pending"
   4) (integer) 1
   5) "idle"
   6) (integer) 1290069
127.0.0.1:6379&gt;

3. View consumption group information

127.0.0.1:6379&gt; xinfo groups stream-key
1) 1) "name"
   2) "g1"
   3) "consumers"
   4) (integer) 2
   5) "pending"
   6) (integer) 2
   7) "last-delivered-id"
   8) "1636362623191-0"
2) 1) "name"
   2) "g2"
   3) "consumers"
......

4. View Stream information

127.0.0.1:6379&gt; xinfo stream stream-key
 1) "length"
 2) (integer) 2
 3) "radix-tree-keys"
 4) (integer) 1
 5) "radix-tree-nodes"
 6) (integer) 2
 7) "last-generated-id"
 8) "1636362623191-0"
 9) "groups"
10) (integer) 4
11) "first-entry"
12) 1) "1636362619125-0"
    2) 1) "aa"
       2) "aa"
13) "last-entry"
14) 1) "1636362623191-0"
    2) 1) "bb"
       2) "bb"
127.0.0.1:6379&gt;

5, Reference documents

1,https://redis.io/topics/streams-intro

2,https://www.runoob.com/redis/redis-stream.html

Posted by devil_online on Wed, 10 Nov 2021 03:05:20 -0800

Programmer Group

Use of Redis Stream type

1, Background

2, Characteristics of Stream type in redis

3, Structure of Stream

4, Stream command

1. XADD adds a message to the end of the Stream

1. Command format

2. Examples

1. Add a piece of data to the stream,

2. Adding data to a stream does not automatically create a stream

3. Manually specify the value of the ID

4. Set a fixed size Stream

1. Specify the exact size of the Stream

2. Specifies the size of the Stream

2. XRANGE viewing messages in a Stream

1. Command format

2. Prepare data

3. Examples

1. Get all data (- and + use)

2. Get the data within the specified id range and close the interval

3. Get the data within the specified id range and open the interval

4. Get all data after a certain millisecond

5. Get a single piece of data

6. Get fixed number of data

3. XREVRANGE reverses the view of messages in the Stream

4. XDEL delete message

1. Command format

2. Prepare data

3. Examples

6. XLEN viewing the length of elements in a Stream

1. Command format

2. Examples

7. XTRIM trims the elements in the Stream

1. Command format

2. Prepare data

3. Examples

1. maxlen exact limit

2. minid fuzzy limit

8. XREAD independent consumption message

1. Command format

2. Prepare data

3. Examples

1. Get the data whose user name is wangwu

2. Get 2 pieces of data

3. Non blocking reading of Stream tail data

4. Blocking the reading of Stream tail data

9. Consumer group related operations

1. Consumer group command

2. Prepare data

3. Create consumer group

1. Create a consumer group to consume from scratch

2. Create a consumer group that consumes the latest message from the Stream

4. Create a consumer group to consume after a message

3. Read message from consumer

4. Read consumer's pending message

5. Transfer consumer messages

6. Some monitoring commands

1. View pending messages for consumers in the consumer group

2. View consumer information in a consumer group

3. View consumption group information

4. View Stream information

5, Reference documents

Hot Keywords