Redis-4.0.14 Cluster installation and deployment
The installation environment of this tutorial is kvm virtual machine and Intranet environment in nat mode. In order to save server resources, this paper uses a single instance Linux host for demonstration. Do not deploy multiple redis instances to a Linux server in the case of large traffic in the production environment (it is not recommended to deploy multiple redis instances to a single machine in the case of small traffic). In the idc machine environment, the concurrency of the network card and the memory type (or memory io type) of the virtual machine should be ensured. After all, redis is a high io service.
Tips: if Redis is a 3-master, 3-slave or above scale cluster, it is recommended to close the bgsave operation on the primary server, instead, it should be done on the slave. When the traffic peak is reduced, bgsave has a delay effect on io throughput.
- Redis download and installation
cd ~ yum install gcc gcc-c++ -y wget http://download.redis.io/releases/redis-4.0.14.tar.gz tar -zxf redis-4.0.14.tar.gz cd redis-4.0.14 cd deps make hiredis jemalloc linenoise lua geohash-int cd .. make PREFIX=/app/sina/redis-9701 install make install
- Add redis cli to the system path
[root@redis-nec001 ~]# cp /app/sina/redis-9701/bin/redis-cli /usr/local/bin/redis-cli [root@redis-nec001 ~]# redis-cli --version redis-cli 4.0.14
- Make multiple copies. The production environment recommends that each instance be configured with a separate server.
cd /app/sina/ mkdir -p redis-9701/conf cp ~/redis-4.0.14/redis.conf ./redis-9701/conf/ cp -arp redis-9701 redis-9702 cp -arp redis-9701 redis-9703 cp -arp redis-9701 redis-9704 cp -arp redis-9701 redis-9705 cp -arp redis-9701 redis-9706 cp -arp redis-9701 redis-9707
- Modify the configuration file redis.conf to modify or open the following items
bind 192.168.100.214 port 9701 daemonize yes pidfile /app/sina/redis-9701/conf/redis_9701.pid logfile "/data/sina/redis-9701/redis.log" dir /data/sina/redis-9701/ maxmemory 4096M cluster-enabled yes cluster-config-file /app/sina/redis-9701/conf/nodes-9701.conf
- Configure ruby environment RVM mode
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB curl -sSL https://get.rvm.io | bash -s stable source /etc/profile.d/rvm.sh rvm install 2.6.4 rvm 2.6.4 --default gem install redis cp ~/redis/redis-4.0.14/src/redis-trib.rb /usr/local/bin/
- Set system kernel parameters
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local cp ~/redis/redis-4.0.14/src/redis-trib.rb /usr/local/bin/ cat /etc/sysctl.conf net.ipv4.ip_forward = 0 net.ipv4.conf.default.accept_source_route = 0 kernel.core_uses_pid = 1 net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 vm.swappiness = 0 net.ipv4.neigh.default.gc_stale_time=120 net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.lo.arp_announce=2 net.core.somaxconn = 262144 net.core.netdev_max_backlog = 262144 net.ipv4.ip_local_port_range = 1024 65000 net.ipv4.tcp_max_tw_buckets = 5000 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_synack_retries = 1 net.ipv4.tcp_syn_retries = 1 net.ipv4.tcp_fin_timeout = 2 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_keepalive_time = 30 net.ipv4.tcp_orphan_retries = 2 kernel.core_pattern = /data/sina/core/core_%e_%p vm.overcommit_memory = 1 kernel.sysrq = 1
- Start redis (start as sina user)
chown -R sina:sina /app/sina/redis* /data/sina/ su - sina /app/sina/redis-9701/bin/redis-server /app/sina/redis-9701/conf/redis.conf /app/sina/redis-9702/bin/redis-server /app/sina/redis-9702/conf/redis.conf /app/sina/redis-9703/bin/redis-server /app/sina/redis-9703/conf/redis.conf /app/sina/redis-9704/bin/redis-server /app/sina/redis-9704/conf/redis.conf /app/sina/redis-9705/bin/redis-server /app/sina/redis-9705/conf/redis.conf /app/sina/redis-9706/bin/redis-server /app/sina/redis-9706/conf/redis.conf /app/sina/redis-9707/bin/redis-server /app/sina/redis-9707/conf/redis.conf
- Cluster creation is divided into three main clusters and three main clusters and three slave clusters (this is also the minimum number of clusters)
redis-trib.rb create --replicas 0 192.168.100.214:9701 192.168.100.214:9702 192.168.100.214:9703 #3 main mode, minimum node cluster, unable to provide high availability. redis-trib.rb create --replicas 1 192.168.100.214:9701 192.168.100.214:9702 192.168.100.214:9703 192.168.100.214:9704 192.168.100.214:9705 192.168.100.214:9706 #In Master-Slave mode, Slave node is the standby node of Master and the read node of data.
- Create a minimum 3 main cluster
[root@redis-nec001 bin]# redis-trib.rb create --replicas 0 192.168.100.214:9701 192.168.100.214:9702 192.168.100.214:9703 >>> Creating cluster >>> Performing hash slots allocation on 3 nodes... Using 3 masters: 192.168.100.214:9701 192.168.100.214:9702 192.168.100.214:9703 M: fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9701 slots:0-5460 (5461 slots) master M: 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 192.168.100.214:9702 slots:5461-10922 (5462 slots) master M: ccf082f6516ec23c1aee891358a3daf47d2b5ca7 192.168.100.214:9703 slots:10923-16383 (5461 slots) master Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.. >>> Performing Cluster Check (using node 192.168.100.214:9701) M: fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9701 slots:0-5460 (5461 slots) master 0 additional replica(s) M: 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 192.168.100.214:9702 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: ccf082f6516ec23c1aee891358a3daf47d2b5ca7 192.168.100.214:9703 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
- View cluster status
[root@redis-nec001 bin]# redis-cli -h 192.168.100.214 -p 9701 -c 192.168.100.214:9701> cluster nodes 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 192.168.100.214:9702@19702 master - 0 1568616352578 2 connected 5461-10922 ccf082f6516ec23c1aee891358a3daf47d2b5ca7 192.168.100.214:9703@19703 master - 0 1568616353579 3 connected 10923-16383 fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9701@19701 myself,master - 0 1568616352000 1 connected 0-5460 192.168.100.214:9701>
- Add slave node to cluster, upgrade to 3 Master 3 slave high availability mode
Redis-trib.rb add node -- slave where -- slave means that the added node is the Master master node of the slave node in the Master ID mode, followed by the ip and port of the slave node to be added, and finally, a Master node is randomly selected to complete the command format. I don't know how to think when designing redis. You have to add an unimportant parameter, but it is necessary.
[root@redis-nec001 sina]# redis-trib.rb add-node --slave --master-id fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9704 192.168.100.214:9701 >>> Adding node 192.168.100.214:9704 to cluster 192.168.100.214:9701 >>> Performing Cluster Check (using node 192.168.100.214:9701) M: fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9701 slots:0-5460 (5461 slots) master 0 additional replica(s) M: 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 192.168.100.214:9702 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: ccf082f6516ec23c1aee891358a3daf47d2b5ca7 192.168.100.214:9703 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 192.168.100.214:9704 to make it join the cluster. Waiting for the cluster to join. >>> Configure node as replica of 192.168.100.214:9701. [OK] New node added correctly.
-
Adding master node to allocate slot flow
The addition configuration of the primary node is slightly more complex. First, the Redis instance stored as empty is added to the cluster, and then the slots are allocated to it'reasonably'.
If the source cluster is a three master cluster, it is better to add another three masters. Each of the original nodes is allocated a general slot to the newly added node.
Namely:
Node 1 allocates half of new node 1
Node 2 allocates two-half of the new nodes
Node 3 allocates half of new node 3
The premise of adding asymmetric mode is that the number of slots in each primary node is the same or nearly the same
Current node slot number = total slot number / cluster node number (including the latest)
Number of slots to be migrated for each node = number of slots for each node of the source - number of slots for the current node - Add master
redis-trib.rb add-node 192.168.100.214:9707 192.168.100.214:9701
- Looking at the slots of each master node, we can see that except for the new nodes, the slots of each master node are 5461.
Slot number of current node (after migration and distribution): 16384 / 4 = 4096
Number of slots to be migrated for the old master node = 5461-4094 = 1365
[root@redis-nec001 sina]# redis-trib.rb check 192.168.100.214:9701 >>> Performing Cluster Check (using node 192.168.100.214:9701) M: fa820855aeebad6551d09d0cd6063aeaefc8f4f9 192.168.100.214:9701 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 62b3ded1a7545f0931611e837cfdbe6dc6fa580c 192.168.100.214:9704 slots: (0 slots) slave replicates fa820855aeebad6551d09d0cd6063aeaefc8f4f9 M: 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 192.168.100.214:9702 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 143e3f136118e62ae6b5a6d64fc21e1fcafee4b4 192.168.100.214:9706 slots: (0 slots) slave replicates ccf082f6516ec23c1aee891358a3daf47d2b5ca7 S: 97147860d71d363059927f21ac92b16b4d17c97e 192.168.100.214:9705 slots: (0 slots) slave replicates 517fd7f65b7e653a91b24aa7a06f1ec360bd8220 M: ccf082f6516ec23c1aee891358a3daf47d2b5ca7 192.168.100.214:9703 slots:10923-16383 (5461 slots) master M: 2c7eb280218234fb6adbd4718c7e21b128f1a938 192.168.100.214:9707 slots: (0 slots) master 0 additional replica(s)
- Slot allocation
redis-trib.rb reshard 192.168.100.214:9707 [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 4096 What is the receiving node ID? 2c7eb280218234fb6adbd4718c7e21b128f1a938 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:fa820855aeebad6551d09d0cd6063aeaefc8f4f9 Source node #2:517fd7f65b7e653a91b24aa7a06f1ec360bd8220 Source node #3:ccf082f6516ec23c1aee891358a3daf47d2b5ca7 Source node #4:done Moving slot 1334 from fa820855aeebad6551d09d0cd6063aeaefc8f4f9 ....... ....... Do you want to proceed with the proposed reshard plan (yes/no)? yes
Allocate the slots on the other two primary nodes in turn, and the process is slightly
After allocation, the slots of each master node are basically the same.
Errors may occur during slot migration. Use the command redis-trib.rb fix to fix them.
Migration error reason
---At that time, the traffic of the master node server was too large, and the cpu load had problems, avoiding the peak value.
---When bgsave is turned on at the host point, bgsave will be turned off, and bgsave will be turned on at the slave node to reduce the pressure on the master node.