Redis cluster construction (Traditional & Docker mode) & cluster capacity expansion & cluster capacity reduction

Keywords: Docker Redis



1, Redis cluster principle

Inter cluster communication mode  

Primary node downtime

Electing a master node from a node

2, RedisCluster installation

Traditional way

Docker mode

3, Cluster expansion

4, Cluster shrinkage

5, Spring boot integrates Redis Cluster

1, Redis cluster principle

Redis cluster does not use consistency hash,   Instead, the concept of hash slot is introduced. The redis cluster has 16384 hash slots. After each key is verified by CRC16, the module of 16384 is taken to determine which slot to place. Each node of the cluster is responsible for a part of the hash slot. For example, the current cluster has three nodes, then:

Node A contains hash slots 0 to 5500.

Node B contains hash slots 5501 to 11000.

Node C contains hash slots 11001 to 16384.

Note: a key value does not correspond to a hash slot. A hash slot can have many key values. Theoretically, a cluster can store many key values.

CRC16 algorithm

static const uint16_t crc16tab[256]= {

uint16_t crc16(const char *buf, int len) {
    int counter;
    uint16_t crc = 0;
    for (counter = 0; counter < len; counter++)
            crc = (crc<<8) ^ crc16tab[((crc>>8) ^ *buf++)&0x00FF];
    return crc;

Inter cluster communication mode  

All cluster nodes establish communication with a binary protocol (cluster bus) through TCP connection. Each node is connected to each other node on the cluster through cluster bus. A gossip protocol is used between nodes to spread cluster information.

Primary node downtime

There are two states of node failure: PFAIL and FAIL. When an active ping packet (A ping packet waiting for its reply after sending) sent by section A to node B in the cluster has been waiting for more than node_ Time out, then A considers that this node is unreachable and marks it as PFAIL. If other nodes also mark node B as PFAIL (node A collects the status information of node B identified by most primary nodes in the cluster through the mission field as PFAIL), then A will mark B as FAIL and tell other reachable nodes that node B has failed.

Electing a master node from a node

The election and promotion of slave nodes are handled by the slave node. The master node will vote which slave node to promote. The election of a slave node occurs when the master node is marked as FAIL by at least one slave node with the necessary conditions to become the master node.

A slave node can initiate an election when the following conditions are met:

  • The master node of the slave node is in the FAIL state.
  • The number of hash slots that this master node is responsible for is not zero.
  • The disconnection of the replication link between the slave node and the master node shall not exceed a given period of time to ensure that the data of the slave node is reliable.
  • If a slave node wants to be elected, the first step should be to increase its currentEpoch (which can be understood as the event version number) count and ask the master nodes to vote.

A failover is broadcast from the node_ AUTH_ The request packet is sent to each master node in the cluster to request votes. Then wait for a reply (up to NODE_TIMEOUT). Once a master node votes for the slave node, it will reply with a FAILOVER_AUTH_ACK, and in NODE_TIMEOUT * 2 can no longer vote for other slave nodes of the same master node during this period.

Once a slave node receives a response from most of the master nodes, it wins the election. Otherwise, if it cannot be in node_ If most primary nodes are accessed within the timeout time, the current election will be interrupted and displayed in node_ After timeout * 4, another slave node attempts to initiate an election.

The slave node does not attempt to initiate an election as soon as the master node enters the FAIL state, but there is a little delay.

The master node receives a request from the slave node to fail_ AUTH_ A request to vote in the form of a request. To grant a vote, the following conditions must be met:

  • In a given period (epoch), a master node can only vote once and refuses to vote in the previous period: each master node has a lastVoteEpoch field. Once the currentEpoch in the authentication request packet is less than the lastVoteEpoch, the master node will refuse to vote again. When a master node actively responds to a voting request, lastVoteEpoch will be updated accordingly.
  • A master node votes for a slave node if and only if the master node of the slave node is marked as FAIL.
  •   If the currentEpoch in the authentication request is smaller than the currentEpoch in the master node, the request will be ignored. Therefore, the response of the master node always carries the currentEpoch consistent with the authentication request. If the same slave node requests to vote again after adding currentEpoch, it is guaranteed that an old delayed reply from the master node will not be accepted by the new round of election.

2, RedisCluster installation

Traditional way

1. Create file

[root@localhost local]# mkdir redis-cluster
[root@localhost redis-cluster]# mkdir 7000
[root@localhost redis-cluster]# mkdir 7001
[root@localhost redis-cluster]# mkdir 7002
[root@localhost redis-cluster]# mkdir 7003
[root@localhost redis-cluster]# mkdir 7004
[root@localhost redis-cluster]# mkdir 7005
#Copy the drop configuration file to the corresponding directory
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7000/ -r
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7001/ -r
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7002/ -r
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7003/ -r
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7004/ -r
[root@localhost redis-cluster]# cp /usr/local/redis-6.2.4/redis.conf /usr/local/redis-cluster/7005/ -r

2. Modify profile

#Comment out
bind -::1
#Background start
daemonize yes 
# Allow external access
protected-mode no 
#Modify the port number from 7000 to 7005
port 7000
#Specify the pid file 7000-7005 for startup
pidfile /var/run/
#Do not lose 7000-7005 quotation marks in the log path
logfile  "usr/local/redis-cluster/7000/redis.log"
#Modify the rdb file for later capacity expansion. Because it is the same virtual machine, it is not necessary to change 7000-7005 if it is not in the same virtual machine
dbfilename dump_7000.rdb
#Change Password
requirepass xiaojie
#The password of the cluster. Otherwise, you need to enter the password when switching nodes
masterauth xiaojie
#Start cluster
cluster-enabled yes 
#Automatic file generation 7000-7005
cluster-config-file nodes-7000.conf

3. Start cluster

Startup script authorization chmod +x  

/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7000/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7001/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7002/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7003/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7004/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis-cluster/7005/redis.conf

4. Create cluster

#No password cluster replicas 1 master-slave node 1:1 assignment
./redis-cli --cluster create  --cluster-replicas 1

#Have a password
./redis-cli --cluster create  --cluster-replicas 1 -a xiaojie

  Enter yes to receive the card slot assignment.

  Seeing this figure shows that the cluster installation is successful.

Cluster nodes command to view cluster nodes


 [root@localhost bin]# ./redis-cli -p 7000 -c -a 'xiaojie'

  OK, the construction is completed.

Docker mode

1. Pull image

[root@localhost bin]# docker pull redis

2. Create file

[root@localhost local]# mkdir docker-redis-cluster 

  3. Create redis-cluster.tmpl file

port ${PORT}
#Cluster password
masterauth xiaojie
#redis password
requirepass xiaojie
#Start cluster
cluster-enabled yes
#configuration file
cluster-config-file nodes.conf

cluster-node-timeout 5000
#If the ip of the cluster communication is accessed through the external network, the public network ip of the server shall be filled in
##Reporting ports of cluster nodes to prevent nat
cluster-announce-port ${PORT}
#Reporting bus port of cluster nodes to prevent nat
cluster-announce-bus-port 1${PORT}
#Open aof
appendonly yes

4. Create data and conf files

[root@localhost docker-redis-cluster]# for port in `seq 9000 9005`; do mkdir -p ./${port}/conf && PORT=${port} envsubst < ./redis-cluster.tmpl > ./${port}/conf/redis.conf && mkdir -p ./${port}/data; done

  5. Create Redis container

[root@localhost docker-redis-cluster]# for port in `seq 9000 9005`; do docker run -d --net=host -v /usr/local/docker-redis-cluster/${port}/conf/redis.conf:/etc/redis/redis.conf -v /usr/local/docker-redis-cluster/${port}/data:/data --restart always --name=redis-${port}  redis redis-server /etc/redis/redis.conf; done

6. Enter any container to create a cluster

#Enter container
[root@localhost docker-redis-cluster]# docker exec -it 9e36f2ab4ae7 bash
#Create cluster
root@localhost:/data# redis-cli --cluster create  --cluster-replicas 1 -a xiaojie

7. Testing

root@localhost:/data# redis-cli -p 9000 -c -a 'xiaojie'  
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.> set docker docker 
-> Redirected to slot [9730] located at
OK> get docker

3, Cluster expansion

Cluster expansion is to add cluster nodes on the basis of the original without restarting the server. Simply put, add a new server, and then distribute some card slots and data from other nodes (if there is data in the card slot, it will be distributed to the new node together with the data).

Add two configuration files, which are available at the source address below

Add master node

#Start the new connection node
[root@localhost bin]# ./redis-server /usr/local/redis-cluster/7006/redis.conf 
[root@localhost bin]# ./redis-server /usr/local/redis-cluster/7007/redis.conf
#Add master node
[root@localhost bin]# /usr/local/redis/bin/redis-cli --cluster add-node -a xiaojie  #- a xiaojie is not required if there is no password


  You can see that the new node has no card slot.

Next, we assign a slave node to the new master node

[root@localhost bin]# ./redis-cli --cluster add-node  --cluster-slave   --cluster-master-id  9493cb0e7ee7b35da0c9078e9a99e5b577de9f1a -a xiaojie 
#The master id is the id corresponding to the 7006 node. You can see that we have added a slave node for 7006


  Enter cluster nodes to check the cluster status, as shown in the figure above. No card slot has been allocated

Assign card slot below

#This client can connect to any node, not limited to 7000
[root@localhost bin]# ./redis-cli --cluster reshard -a xiaojie

  How many slots do you want to move? 16384 / 4 = 4096, so we want to move 4096 card slots to 7006 node, and then distribute them to 7006 from the other three main nodes (there are no card slots on the slave node). So enter 4096


  Enter the corresponding id. here, we select all, and then select yes. Just wait until the card slot is allocated.

It can be seen from the above figure on the right that the card slot has been allocated to us.  

4, Cluster shrinkage

Cluster capacity reduction is just the opposite of capacity expansion, that is, removing service nodes without restarting the server.

  We choose to give all 7006 card slots to 7001. Note: when removing a node, make sure that all card slots have been removed to other primary nodes.

[root@localhost bin]# ./redis-cli --cluster  reshard  --cluster-from   9493cb0e7ee7b35da0c9078e9a99e5b577de9f1a -a xiaojie  --cluster-to d5ac3163a51a0b7e70e777d8ff137a4ba901d611  --cluster-slots -a xiaojie

5, Spring boot integrates Redis Cluster

    time-zone: GMT+8
    date-format: yyyy-MM-dd HH:mm:ss
    name: my-test
    url: jdbc:mysql://
    username: root
    password: root
    driver-class-name: com.mysql.cj.jdbc.Driver
      nodes: #I am connected to the cluster installed by Docker
    password: xiaojie
    connect-timeout: 5000
#    password: xiaojie #This password must be added before you can connect to the redis server. If no password is configured, it is not required
#    connect-timeout: 5000
#    database: 0
#    sentinel:
#      master: mymaster
#      nodes:,,
#      password: xiaojie #This password is the password of sentinel. If requirepass is not configured in sentinel.conf, it is not required

Complete code and cluster configuration file: 

reference resources:

Redis cluster spec - redis Chinese data station - redis China user group (CRUG)

docker deploys redis Cluster - bridgehead foot washing city - blog Park

Posted by alfieshooter on Fri, 10 Sep 2021 17:02:32 -0700