This series of articles outlines five common working modes of LVS, followed by a brief description of the working principle of LVS.
IPVSADM and IPVS
LVS is responsible for four-tier load balancing scheduling. When we define the scheduling rules of clusters, we need to use a management tool, ipvsadm. The rules defined by ipvsadm are ultimately implemented through ipvs, that is to say, ipvsadm is the management tool of ipvs, ipvsadm works in user space and IPVS works in kernel space. The relationship between them is similar to that between iptables and netfliter.
IPVS supports the TCP/UDP protocol. It establishes a state tracking mechanism for all data packets in a TCP connection from the TCP SYNC packet to ensure that all data packets in a TCP connection can reach the same back end. So IPVS is based on the control and management of TCP state machine and only perceives the TCP header but does not view the TCP payload; therefore, there is another assumption for the IPVS back-end server cluster, that is, all backends have the same application-level service function, but because IPVS can set weight to the back-end, so each back-end serves. The capabilities of business can vary.
The history of IPVS (IP Virtual Server):
- As early as the 2.2 kernel, IPVS has appeared in the form of kernel patches.
- Since version 2.4.23, ipvs software is a collection of kernel patches that are merged into the common version of the linux kernel.
- Since 2.4.24, IPVS has become part of the official linux standard kernel and exists as a module of Netfilter.
When Client accesses the service, VIP and its corresponding ports will be accessed, so the request message received by LVS first passes through the PREROUTING chain and then makes routing judgment. Because the DIP at this time is VIP, the request message will pass through the INPUT chain for the device IP (which is configured on the network card interface). At this time, if IPVS finds the VIP and the route of the message accessed by the IPVS, the request message will pass through the INPUT chain. When the port matches the defined rules, IPVS will send messages directly to POSTROUTING chain according to the defined rules and algorithms, and finally the messages will be sent from LVS to the back-end RS.
IPVSADM usage
Let's take the DR mode of LVS as an example. After installing ipvsadm, we execute the following commands on the dispatcher
ipvsadm -A -t VIP:port -s rr ipvsadm -a -t VIP:port -r RIP1 -g ipvsadm -a -t VIP:port -r RIP2 -g //The configuration in this case is as follows //ipvsadm -A -t 192.168.0.100:80 -s rr //ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.4 -g //ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.5 -g
ipvsadm -A -t VIP:port -s rr
- Option A adds a virtual server record to create an LVS cluster
- t identifies the created LVS cluster service as tcp protocol, and VIP:port identifies the cluster service IP and port number.
- s Scheduling scheduling means that the following parameters are specified scheduling algorithms, including rr, wrr, lc, wlc, lblc, lblcr, dh, sh, sed, nq, default to wlc, where RR is abbreviated as Round-Robin, representing polling algorithm
So after creating a LVS service cluster, we need to set up specific back-end RS, scheduling algorithm and other information.
ipvsadm -a -t VIP:port -r RIP -g
- The a option means adding an RS to the cluster
- t Ibid.
- r is used to specify the RIP to be added (RS corresponding IP)
- g denotes that LVS operates in DR mode, - i in TUN mode and - m in NAT mode.
After configuring, we check the effective rules through ipvsadm-Ln, but it should be noted that if the rules are not saved in advance, the configuration will disappear after the device restart, so we need to save the command with service ipvsadm save after configuring, and reload the rules with service ipvsadm reload after restarting. In addition, LVS rules can be saved in specified files with - S and overloaded with - R.
ipvsadm -S > file ipvsadm -R < file
The IPVSADM command can refer to this link
LVS-DR example
To simulate the cluster environment, we have four virtual machines: client, LVS server, Real Server 1, Real Server 2.
- Client IP: 192.168.0.2
- LVS IP: 192.168.0.3
- RealServer1 IP: 192.168.0.4
- RealServer2 IP: 192.168.0.5
- Vitual IP: 192.168.0.100 (virtual IP)
LVS configuration
First, configure LVS. After installing ipvsadm in yum, the system has not loaded the ipvsadm module into the system at this time. We need to execute the ipvsadm command to load it or modprobe ip_vs.
After configuring the rules, you can view the effective cluster rules through ipvsadm-ln. Of course, in order to restart and continue to use the currently configured rules, you can save them by-S.
Because VIP needs to be bound to the network card of LVS and broadcasted by ARP (for external, the corresponding device of VIP is LVS, and the corresponding ARP request can only be responded by LVS), we add VIP 192.168.0.100/24 to ens33:0.
[root@LVS ~]# yum install ipvsadm -y [root@LVS ~]# lsmod | grep ip_vs [root@LVS ~]# //ipvsadm is not enabled at this time [root@LVS ~]# ipvsadm -A -t 192.168.0.100:80 -s rr [root@LVS ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.4 -g [root@LVS ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.5 -g [root@LVS ~]# lsmod | grep ip_vs ip_vs_rr 12600 1 ip_vs 145497 3 ip_vs_rr nf_conntrack 133095 7 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack //ipvsadm is enabled [root@LVS ~]# ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.100:80 rr -> 192.168.0.4:80 Route 1 0 0 -> 192.168.0.5:80 Route 1 0 0 //ipvsadm rules come into force [root@LVS ~]# ifconfig ens33:0 192.168.0.100/32 up
Real Server configuration
First, deploy the http service, which uses nginx. Because the destination IP of receiving data packet is VIP, VIP needs to be added to lo, because RS VIP is not used for communication, so it is necessary to set / 32 bit mask here!
[root@RealServer1 ~]# rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm [root@RealServer1 ~]# yum install nginx -y [root@RealServer1 ~]# nginx //start nginx [root@RealServer1 ~]# vi /usr/share/nginx/html/index.html //Change to Real Server 1 192.168.0.4 [root@RealServer1 ~]# curl 127.0.0.1 RealServer1 192.168.0.4 //So far, http services have been deployed [root@RealServer1 ~]# ifconfig lo:0 192.168.0.100/32 up //Add VIP to lo:0 with a mask of / 32
At this point, I mistakenly think that all of them are fully configured and the cluster can serve normally. However, when ARP asks for VIP, LVS, RS1 and RS2 will respond, resulting in the failure of load balancing.
[root@Client ~]# arping -I ens33 192.168.0.100 ARPING 192.168.0.100 from 192.168.0.3 ens33 Unicast reply from 192.168.0.100 [00:0C:29:AF:6B:F7] 1.778ms Unicast reply from 192.168.0.100 [00:0C:29:AC:67:31] 1.852ms Unicast reply from 192.168.0.100 [00:0C:29:BD:38:DA] 1.860ms Unicast reply from 192.168.0.100 [00:0C:29:BD:38:DA] 1.860ms Unicast reply from 192.168.0.100 [00:0C:29:BD:38:DA] 1.860ms //Three devices will be found to respond to ARP requests, eventually to 00:0C:29:BD:38:DA (RS1) [root@Client ~]# arping -I ens33 192.168.0.2 ARPING 192.168.0.2 from 192.168.0.3 ens33 Unicast reply from 192.168.0.2 [00:0C:29:AF:6B:F7] 1.500ms [root@Client ~]# arping -I ens33 192.168.0.4 ARPING 192.168.0.4 from 192.168.0.3 ens33 Unicast reply from 192.168.0.4 [00:0C:29:BD:38:DA] 1.609ms [root@Client ~]# arping -I ens33 192.168.0.5 ARPING 192.168.0.5 from 192.168.0.3 ens33 Unicast reply from 192.168.0.5 [00:0C:29:AC:67:31] 1.603ms //The three devices are LVS, RS1 and RS2 respectively.
Because the VIP is configured on the lo of RS, and because arp_ignore defaults to 0, the device responds to the ARP request corresponding to the VIP tacitly. Configuration Note Reference arp_ignore and arp_announce of Linux kernel parameters
- arp_ignore
In DR mode, each RS adds VIP to lo. Because arp_ignore defaults to 0, each RS will respond to the corresponding ARP request of VIP, at this time Client can not correctly obtain the network card mac where the VIP is located on LVS. Although the request message arrives on one of the RS, RS can respond normally, but because the message bypasses LVS and does not play the role of load balancing, it is easy to appear bottlenecks when the traffic is large.
So in DR mode, the arp_ignore parameter is required to be configured as 1.
- arp_announce
Each device maintains its own ARP table. When it receives an ARP request with an unknown IP address, it adds IP and MAC records to the local ARP table. When it receives an ARP request with a known IP address (the address already recorded in the ARP table), it refreshes its ARP table according to the source MAC in the ARP request.
Therefore, in DR mode, the arp_announce parameter is required to be configured as 2.
[root@RealServer1 ~]# echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore [root@RealServer1 ~]# echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore [root@RealServer1 ~]# echo "2">/proc/sys/net/ipv4/conf/all/arp_announce [root@RealServer1 ~]# echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
Client test
After configuring the arp_ignore and arp_announce of Real Server 1, Real Server 2, test the service on Client
[root@Client ~]# arping -I ens33 192.168.0.100 ARPING 192.168.0.100 from 192.168.0.3 ens33 Unicast reply from 192.168.0.100 [00:0C:29:AF:6B:F7] 1.301ms Unicast reply from 192.168.0.100 [00:0C:29:AF:6B:F7] 0.827ms Unicast reply from 192.168.0.100 [00:0C:29:AF:6B:F7] 0.887ms Unicast reply from 192.168.0.100 [00:0C:29:AF:6B:F7] 0.801ms //Response to ARP requests by LVS [root@Client ~]# curl 192.168.0.100 RealServer2 192.168.0.5 [root@Client ~]# curl 192.168.0.100 RealServer1 192.168.0.4 [root@Client ~]# curl 192.168.0.100 RealServer2 192.168.0.5 [root@Client ~]# curl 192.168.0.100 RealServer1 192.168.0.4 //Response requests are polled by Real Server 1 and Real Server 2 1:1
So far we have manually configured a most basic DR-LVS cluster, but the shortcomings are also obvious:
- More configuration, maintenance is not easy
- ipvs rules are fixed, and if the back-end RS failures are still allocated, there is no back-end detection
- Single LVS with high single point failure probability
So follow-up introduction keepalived-LVS cluster to solve the above problems