1. Introduction to keepalived
Keepalived software was originally designed for LVS load balancing software to manage and monitor the status of each service node in LVS cluster system. Later, it added VRRP function that can realize high availability. Therefore, in addition to managing LVS software, keepalived can also be used as high availability solution software for other services (such as Nginx, Haproxy, MySQL, etc.).
Keepalived software mainly realizes high availability through VRRP protocol. VRRP is the abbreviation of virtual router redundancy protocol. The purpose of VRRP is to solve the problem of single point of failure of static routing. It can ensure that the whole network can run continuously when individual nodes are down.
Therefore, on the one hand, kept has the function of configuring and managing LVS, and also has the function of health inspection for the nodes under LVS. On the other hand, it can also realize the high availability of system network services.
Official website: http://www.keepalived.org/
1.2 functions of keepalived
- Manage LVS load balancing software
- Realize the health check of LVS cluster nodes
- High availability as a system network service (failover)
1.3 principle of keepalived high availability failover
Failover between Keepalived high availability services is realized through VRRP.
When the Keepalived service works normally, the primary Master node will continuously send messages to the standby node (multicast) The heartbeat message is used to tell the standby Backup node that it is still alive. When the primary Master node fails, it cannot send the heartbeat message, so the standby node cannot continue to detect the heartbeat from the primary Master node, so it calls its own takeover program to take over the IP resources and services of the primary Master node. When the primary Master node recovers, the standby Backup node returns The IP resources and services taken over by the primary node in case of failure will be released and restored to the original standby role.
-
VRRP: its full name is Virtual Router Redundancy Protocol, and its Chinese name is virtual routing redundancy protocol. The emergence of VRRP is to solve the problem of static single point of failure. VRRP gives the routing task to a VRRP router through a campaign mechanism.
-
VRRP uses IP Multicast (default multicast address (224.0_0.18)) to realize the communication between high availability pairs.
-
During operation, the master node contracts and the standby node receives the packets. When the standby node cannot receive the data packets sent by the master node, it starts the takeover program to take over the open source of the master node. There can be multiple standby nodes competing through priority, but generally there are one pair in the operation and maintenance of the Keepalived system.
-
VRRP uses encryption protocol to encrypt data, but Keepalived officials still recommend configuring authentication type and password in plaintext.
1.4 working principle of kept
Keepalived high availability communicates through VRRP. VRRP determines the active and standby through the election mechanism. The primary node has higher priority than the standby. Therefore, when working, the primary node will give priority to obtaining all resources, and the standby node is in the waiting state. When the primary node hangs up, the standby node will take over the resources of the primary node, and then provide services on behalf of the primary node.
Between Keepalived services, only the master server will always send VRRP broadcast packets and tell the standby server that it is still alive. At this time, the standby server will not occupy the master. When the master is unavailable, that is, the standby server cannot monitor the broadcast packets sent by the master, it will start relevant services to take over resources to ensure business continuity. The fastest takeover speed can be less than 1 second.
keepalived deploy high availability httpd
Environmental description:
system information | host name | IP |
---|---|---|
centos7 | master | 192.168.216.204 |
centos7 | slave | 192.168.216.179 |
The VIP (virtual ip) address is tentatively 192.168.216.250. For details, see previous articles LVS
- Install master keepalived
//Turn off the firewall, selinux [root@master ~]# systemctl disable --now firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@master ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) [root@master ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [root@master ~]# setenforce 0 [root@master ~]# getenforce Permissive //Configure network source [root@master ~]# curl -o /etc/yum.repos.d/CentOS7-Base-163.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1572 100 1572 0 0 9684 0 --:--:-- --:--:-- --:--:-- 9703 [root@master ~]# sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS7-Base-163.repo [root@master ~]# sed -i 's/^enabled=.*/enabled=1/g' /etc/yum.repos.d/CentOS7-Base-163.repo //Install related components [root@master ~]# yum -y install epel-release vim wget gcc gcc-c++ //Install keepalived [root@master ~]# yum -y install keepalived [root@master ~]# rpm -ql keepalived /etc/keepalived //configure directory /etc/keepalived/keepalived.conf //Master profile /etc/sysconfig/keepalived /usr/bin/genhash /usr/lib/systemd/system/keepalived.service //Service control document /usr/libexec/keepalived /usr/sbin/keepalived //Command storage location
Use the same method to install keepalived on the standby server
- Installed standby keepalived
//Turn off the firewall, selinux [root@slave ~]# systemctl disable --now firewalld.service Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@slave ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/selinux/config [root@slave ~]# setenforce 0 //Configure network source [root@slave ~]# curl -o /etc/yum.repos.d/CentOS7-Base-163.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo [root@slave ~]# sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS7-Base-163.repo [root@slave ~]# sed -i 's/^enabled=.*/enabled=1/g' /etc/yum.repos.d/CentOS7-Base-163.repo [root@slave ~]# yum -y install epel-release vim wget gcc gcc-c++ //Install keepalived [root@slave ~]# yum -y install keepalived
Install httpd on the active and standby machines respectively
- Installing httpd on the master
//Install httpd [root@master ~]# yum -y install httpd //Start and set the startup self start [root@master ~]# systemctl enable --now httpd Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service. [root@master ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:22 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 :::80 :::* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* //Configure web site [root@master ~]# echo 'this is master' > /var/www/html/index.html [root@master ~]# curl 192.168.216.204 this is master
- Installing httpd on slave
[root@slave ~]# yum -y install httpd [root@slave ~]# systemctl enable --now httpd Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service. [root@slave ~]# echo 'this is slave' > /var/www/html/index.html [root@slave ~]# curl 192.168.216.179 this is slave [root@slave ~]# curl 192.168.216.204 this is master
- Configure master keepalived
[root@master ~]# vim /etc/keepalived/keepalived.conf [root@master ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id lb01 } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass lll } virtual_ipaddress { 192.168.216.250 } } virtual_server 192.168.216.250 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.216.204 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.216.179 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } //Restart for configuration to take effect [root@master ~]# systemctl restart keepalived [root@master ~]# systemctl enable keepalived Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
- Configure standby keepalived
[root@slave ~]# vim /etc/keepalived/keepalived.conf [root@slave ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id lb02 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass lll } virtual_ipaddress { 192.168.216.250 } } virtual_server 192.168.216.250 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.216.204 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.216.179 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } //Make configuration effective [root@slave ~]# systemctl restart keepalived [root@slave ~]# systemctl enable keepalived Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service. //View VIP [root@master ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:82:b6:d0 brd ff:ff:ff:ff:ff:ff inet 192.168.216.204/24 brd 192.168.216.255 scope global noprefixroute dynamic ens33 valid_lft 1486sec preferred_lft 1486sec inet 192.168.216.250/32 scope global ens33 //VIP here valid_lft forever preferred_lft forever inet6 fe80::78af:cdcb:c379:1773/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@slave ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:b8:1e:94 brd ff:ff:ff:ff:ff:ff inet 192.168.216.179/24 brd 192.168.216.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feb8:1e94/64 scope link valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:b6:a9:b9 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:b6:a9:b9 brd ff:ff:ff:ff:ff:ff
Modify kernel parameters and enable listening VIP function
//Modify kernel parameters on master [root@master ~]# echo 'net.ipv4.ip_nonlocal_bind = 1' >>/etc/sysctl.conf [root@master ~]# sysctl -p net.ipv4.ip_nonlocal_bind = 1 [root@master ~]# cat /proc/sys/net/ipv4/ip_nonlocal_bind 1 //Modify kernel parameters on slave [root@slave ~]# echo 'net.ipv4.ip_nonlocal_bind = 1' >>/etc/sysctl.conf [root@slave ~]# sysctl -p net.ipv4.ip_nonlocal_bind = 1 [root@slave ~]# cat /proc/sys/net/ipv4/ip_nonlocal_bind 1
Let keepalived monitor the status of httpd load balancer through script
//Script on master [root@master ~]# mkdir /scripts [root@master scripts]# vim check_m.sh [root@master scripts]# cat check_m.sh #!/bin/bash httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l) if [ $httpd_status -lt 1 ];then systemctl stop keepalived fi [root@master scripts]# chmod +x check_m.sh [root@master scripts]# vim notify.sh [root@master scripts]# cat notify.sh #!/bin/bash VIP=$2 sendmail (){ subject="${VIP}'s server keepalived state is translate" content="`date +'%F %T'`: `hostname`'s state change to master" echo $content | mail -s "$subject" 593582553@qq.com } case "$1" in master) httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l) if [ $httpd_status -lt 1 ];then systemctl start httpd fi sendmail ;; backup) httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l) if [ $httpd_status -gt 0 ];then systemctl stop httpd fi ;; *) echo "Usage:$0 master|backup VIP" ;; esac [root@master scripts]# chmod +x notify.sh [root@master scripts]# ll Total consumption 8 -rwxr-xr-x. 1 root root 142 10 June 22 00:27 check_m.sh -rwxr-xr-x. 1 root root 662 10 June 22 00:28 notify.sh
//Script on slave [root@slave ~]# mkdir /scripts [root@slave ~]# cd /scripts/ [root@slave scripts]# vim notify.sh [root@slave scripts]# cat notify.sh #!/bin/bash VIP=$2 sendmail (){ subject="${VIP}'s server keepalived state is translate" content="`date +'%F %T'`: `hostname`'s state change to master" echo $content | mail -s "$subject" 1344933566@qq.com } case "$1" in master) httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l) if [ $httpd_status -lt 1 ];then systemctl start httpd fi sendmail ;; backup) httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l) if [ $httpd_status -gt 0 ];then systemctl stop httpd fi ;; *) echo "Usage:$0 master|backup VIP" ;; esac [root@slave scripts]# chmod +x notify.sh [root@slave scripts]# ll Total consumption 4 -rwxr-xr-x. 1 root root 663 10 September 21:32 notify.sh
- Configure the configuration of keepalived to join the monitoring script
//Configure master keepalived [root@master scripts]# vim /etc/keepalived/keepalived.conf [root@master scripts]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id lb01 ## It is very important to identify the hostname of this machine } vrrp_script httpd_check { script "/scripts/check_m.sh" ## Script location to execute interval 1 ## Detection interval weight -20 ## If the condition holds, the weight is reduced by 20 } vrrp_instance VI_1 { state MASTER ## It is very important to identify the host and BACKUP the standby interface ens33 ## Very important, network card name virtual_router_id 52 ## Very important. Virtual route ID number (primary and standby nodes should be the same) priority 100 ## Priority (0-254). Generally, the priority of the host is greater than that of the standby advert_int 1 ## The sending interval of active and standby information must be the same between the two nodes. The default is 1 second authentication { auth_type PASS auth_pass lll Certificate matching, setting authentication type and password, MASTER and BACKUP You must use the same password to communicate properly } virtual_ipaddress { 192.168.216.250 ##Virtual ip, multiple can be specified } track_script { ## Script to check the health of haproxy httpd_check } notify_master "/scripts/notify.sh master 192.168.216.250" notify_backup "/scripts/notify.sh backup 192.168.216.250" } virtual_server 192.168.216.250 80 { ####Detailed configuration of virtual ip delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.216.179 80 { ## ## Real ip address of the machine weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.216.204 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@master scripts]# systemctl restart keepalived
- Configure standby keepalived
Backup does not need to check whether httpd is normal. Start httpd when upgrading to MASTER and close it when demoting to backup
[root@slave scripts]# vim /etc/keepalived/keepalived.conf [root@slave scripts]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id lb02 } vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 52 priority 90 advert_int 1 authentication { auth_type PASS auth_pass lll } virtual_ipaddress { 192.168.216.250 } notify_master "/scripts/notify.sh master 192.168.216.250" notify_backup "/scripts/notify.sh backup 192.168.216.250" } virtual_server 192.168.216.250 80 { delay_loop 6 lb_algo rr lb_kind DR persistence_timeout 50 protocol TCP real_server 192.168.216.204 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.216.179 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@slave scripts]# systemctl restart keepalived
Visit the web page to see
Simulate primary server downtime
[root@master scripts]# systemctl stop keepalived [root@master scripts]# systemctl stop httpd
//The backup server receives the mail and the vip is transferred [root@slave scripts]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:b8:1e:94 brd ff:ff:ff:ff:ff:ff inet 192.168.216.179/24 brd 192.168.216.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.216.250/32 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feb8:1e94/64 scope link valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:b6:a9:b9 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:b6:a9:b9 brd ff:ff:ff:ff:ff:ff You are /var/spool/mail/root Mail in [root@slave scripts]# curl 192.168.216.250 this is slave You are /var/spool/mail/root New messages in