In the previous chapter, the blogger introduced the architecture of the mainalived + nginx (backup) high-availability system for large-scale Internet projects. I believe you should read the blogger's article and have a certain understanding of the mainalived / nginx technology. In this section, the blogger will share the related technology and configuration process of the mainalived + nginx (multi-master and multi-life) high-availability system.
Since the previous article has introduced the installation process of keeping alived and nginx, this section will not repeat the burden, but will start talking about the live configuration and automated script monitoring.
Configuration steps (taking the three main configurations of three nginx servers as an example):
First, install software such as keepalived and nginx according to the previous chapter
2. Keeping alived to modify configuration files
(2.1) First server
keepalived.config configuration file (directory/etc/keepalived/keepalived.conf)
! Configuration File for keepalived global_defs { } vrrp_script chk_nginx { #script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1" script "/usr/local/keepalived/sbin/check_ng_pid.sh" interval 1 #Execute the above script every 1 second to check the user's program ngnix weight -10 } vrrp_instance VI_1 { state MASTER #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 52 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 200 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.191/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_2 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 53 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 150 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.192/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_3 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 54 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 100 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.193/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" }
(2.2) Second server
keepalived.config configuration file (directory/etc/keepalived/keepalived.conf)
! Configuration File for keepalived global_defs { } vrrp_script chk_nginx { #script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1" script "/usr/local/keepalived/sbin/check_ng_pid.sh" interval 1 #Execute the above script every 1 second to check the user's program ngnix weight -10 } vrrp_instance VI_1 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 52 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 100 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.191/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_2 { state MASTER #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 53 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 200 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.192/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_3 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 54 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 150 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.193/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" }
(2.3) Third server
keepalived.config configuration file (directory/etc/keepalived/keepalived.conf)
! Configuration File for keepalived global_defs { } vrrp_script chk_nginx { #script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1" script "/usr/local/keepalived/sbin/check_ng_pid.sh" interval 1 #Execute the above script every 1 second to check the user's program ngnix weight -10 } vrrp_instance VI_1 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 52 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 150 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.191/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_2 { state BACKUP #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 53 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 100 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.192/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" } vrrp_instance VI_3 { state MASTER #Specify A node as the primary node and BACKUP as the standby node. interface eth0 #Network Interface Binding Virtual IP virtual_router_id 54 #The VRRP group name, which is consistent in the same instance, is unique throughout the VRRP to indicate that each node belongs to the same VRRP group. priority 200 #Priority of primary node (between 1 and 254), standby node must have lower priority than primary node. advert_int 1 #Multicast message sending interval, two nodes must be set the same authentication { #To set up validation information, the two nodes must be consistent auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { #Specify virtual IP, consistent in the same instance, unique in the entire vrrp 192.168.29.193/24 #If the two nginx IPS are 192.168.33.61,... 62 respectively, then the virtual ip here is the same segment as the two. } notify_master "/usr/local/keepalived/sbin/notify.sh master" notify_backup "/usr/local/keepalived/sbin/notify.sh backup" notify_fault "/usr/local/keepalived/sbin/notify.sh fault" }
(2.4) scripts required on each server
Notification script configuration / usr/local/keepalived/sbin/notify.sh
#!/bin/bash case "$1" in master) /usr/local/nginx/sbin/nginx exit 0 ;; backup) /usr/local/nginx/sbin/nginx -s stop /usr/local/nginx/sbin/nginx exit 0 ;; fault) /usr/local/nginx/sbin/nginx -s stop exit 0 ;; *) echo 'Usage: notify.sh {master|backup|fault}' exit 1 ;; esac
nginx service check script / usr/local/keepalived/sbin/check_ng_pid.sh
#Monitor the nginx process and start nginx if the main nginx process does not exist # If the nginx process does not exist after 5s, kill the keepalived process to prevent nginx from not running the mainframe's keepalived and take over the virtual IP #!/bin/bash SERVER=127.0.0.1 PASSWORD=hadoop #Getting ssh public key get_rsa(){ expect -c "set timeout -1; spawn ssh-keygen -t rsa; expect { *Enter* {send -- \r;exp_continue;} {Overwrite (y/n)*} {send -- n\r;exp_continue} eof {exit 0;} }"; } auto_ssh_copy_id() { expect -c "set timeout -1; spawn ssh-copy-id $1; expect { *(yes/no)* {send -- yes\r;exp_continue;} *assword:* {send -- $2\r;exp_continue;} # *ERROR: No identities found* {get_rsa;ssh_copy_id_to_all ;exp_continue; } eof {exit 0;} }"; } ssh_copy_id_to_all() { auto_ssh_copy_id $SERVER $PASSWORD } #Execute delay_stop.sh on the remote host execute_sh(){ expect -c "set timeout -1; spawn ssh root@$SERVER nohup /usr/local/keepalived/sbin/delay_stop.sh > /dev/null 2>&1 & expect { *(yes/no)* {send -- yes\r;exp_continue;} *password:* {send -- $1\r;exp_continue;} eof {exit 0;} }"; } get_rsa ssh_copy_id_to_all c1=`netstat -antp |grep -v grep |grep nginx |wc -l` if [ $c1 -eq 0 ]; then #/usr/local/nginx/sbin/nginx #sleep 2 c2=`netstat -antp |grep -v grep |grep nginx |wc -l` if [ $c2 -eq 0 ]; then execute_sh $PASSWORD # ssh root@$SERVER > /dev/null 2>&1 <<eeooff #nohup service keepalived stop & # nohup /usr/local/keepalived/sbin/delay_stop.sh > /dev/null 2>&1 & # exit #eeooff exit 0 # ssh root@$SERVER "/etc/init.d/keepalived stop" > /usr/local/keepalived/sbin/a.txt # service keepalived stop # killall keepalived # /etc/init.d/keepalived stop #ps -ef | grep keepalived | grep -v grep | awk '{print $8}' | xargs kill else exit 0 fi else exit 0 fi
Close the keepalived service to implement vip drifting scripts
#bin/bash sleep 3 ssh root@127.0.0.1 service keepalived stop #service keepalived stop
(2.5) Configured, check whether openssh-server, openssh-clients, expect (using commands in automation scripts) have been installed
#Install ssh rpm -qa|grep openssh See openssh Has the component been installed? yum list|grep openssh list yum Installable in the library openssh software package yum install -y openssh-server Use yum install yum install -y openssh-clients Use yum install #Install expect rpm -qa|grep expect yum list|grep expect yum install -y expect
(2.6) Start the ssh service, command service sshd start, check whether root users are available to connect
ssh root@127.0.0.1 If the reporting authority is limited, vi/etc/ssh/sshd_config needs to be changed to remove the comment on PermitRootLogin yes line ""
Note: Bloggers are lazy here and use root users directly. In production environments, special users are usually used to execute automated scripts.
(2.7) Testing the availability of keepalived+nginx high-availability multi-master multi-live cluster
a. Stop the firewall on three servers: service iptables stop.
b. Start the keep alived command on three servers (keep alived will automatically start nginx): service keep alived start
c. Check whether the address vip of the three server network cards is properly bound
d. Close the nginx service on one of the servers and execute the command: kill all nginx to see if the network card vip binding changes.
vip is no longer on this server.
e. Visit the virtual VIP of the vip-elegant machine: http://192.168.29.191 and find that nginx can be accessed, indicating that VIP has been elegant to other machines (can be viewed through the ip addr command)
f. Check the keepalived process of the server that shuts down nginx. The keepalived process has been automatically shut down and drifting is completed.
g. Restart the closed keepalived and check whether the vip binding returns to the local machine
Check the vip virtual machine for nginx accessibility: http://192.168.29.191
i.vip drift, reprints are completed smoothly and the tutorial is over
Summary: Because bloggers in this chapter spend too much time recording tutorials, they don't configure firewalls here. Bloggers encounter many problems in scripting, which are solved by shell script debugging tools: sh-vx check_ng_pid.sh; if you have problems in configuration, you can execute the shell debugging command once. This is the whole process of nginx multi-master multi-live architecture. If you think the blogger's article is good, please comment; if you are interested in other server technology or blogger himself, please pay attention to the blogger's blog, and welcome to communicate with the blogger at any time.