Preface
For websites with large visits, with the increase of traffic, a single server has been unable to process all requests. At this time, multiple servers are required to shunt a large number of requests, that is, load balancing. If load balancing is achieved, a server (not just one) must be deployed at the entrance of the website to distribute these requests, which is a reverse proxy. Because the reverse proxy server is the entrance of the website, its load pressure is high and vulnerable to attack, and there is a risk of single point failure, so we need a highly available solution to realize that when one reverse proxy server goes down, another server will take over the service automatically. Based on the above requirements, we use HAProxy and KeepAlived to build a highly available reverse proxy system.
introduce
HAProxy It is a high-performance proxy server. It can provide 7-tier and 4-tier proxies. It has health check, load balancing and other characteristics. It has excellent performance. It includes many well-known Internet companies such as Twitter, Reddit, Stack Overflow and GitHub. Use.
KeepAlived It is a high availability scheme, which is realized by VIP (virtual IP) and heartbeat detection. The principle is that there is a set of servers (two servers), which give Master and Backup two roles respectively. By default, Master will bind VIP to its own network card to provide services to the outside world. Master,Backup will send heartbeat packets to each other at a certain time interval to detect each other's state. This time interval is usually 2 seconds. If Backup finds that Master is down, then Backup will send ARP packets to the gateway and bind VIP to its own network card. Backup will provide services to the outside world and realize automatic fault transfer. When Master recovers, it will take over the service again. Business.
Environmental Science
OS: CentOS Linux release 6.0 (Final) 2.6.32-71.29.1.el6.x86_64
HAProxy: 1.4.18
KeepAlived: 1.2.2
VIP: 192.168.1.99
M: 192.168.1.222
S: 192.168.1.189
Framework
192.168.1.99
+-----------VIP----------+
| |
| |
Master Backup
192.168.1.189 192.168.1.222
+----------+ +----------+
| HAProxy | | HAProxy |
|keepalived| |keepalived|
+----------+ +----------+
|
v
+--------+---------+
| | |
| | |
v v v
+------+ +------+ +------+
| WEB1 | | WEB2 | | WEB3 |
+------+ +------+ +------+
Install HAProxy
Install pcre
$ yum install pcre $ wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.18.tar.gz $ tar -zxvf haproxy-1.4.18.tar.gz $ cd haproxy-1.4.18
Note the compilation parameters:
TARGET is the kernel version of your system ARCH specifies whether the system is 32-bit or 64-bit.
CPU=native: use the build machine's specific processor optimizations
See README in the source code for more compilation parameters
$ make TARGET=linux26 ARCH=x86_64 USE_PCRE=1 CPU=native $ make install
Configuration file/etc/haproxy.cfg
global
log 127.0.0.1 local3
maxconn 20000
uid 535 #uid and gid are configured according to the actual situation
gid 520
chroot /var/chroot/haproxy
daemon
nbproc 1
defaults
log 127.0.0.1 local3
mode http
option httplog
option httpclose
option dontlognull
option forwardfor
retries 2
balance roundrobin
stats uri /haproxy-stats
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http-in
bind *:80
default_backend pool1
backend pool1
option httpchk HEAD / HTTP/1.0
stats refresh 2
server WEB1 192.168.1.189:81 weight 3 maxconn 10000 check
server WEB2 192.168.1.222:81 weight 3 maxconn 10000 check
Check the status of HAProxy: http://192.168.1.99/haproxy-stats, which displays the status of HAProxy itself and the back-end server.
Journal
haproxy sends log records to syslog server (under CentOS6 is rsyslogd, UDP514 port), edits / etc/rsyslog.conf file, and adds the following:
$ModLoad imudp $UDPServerRun 514 $UDPServerAddress 127.0.0.1 local3.* /var/log/haproxy.log
Restart rsyslog
$ /etc/init.d/rsyslog restart
Automated rotation log, edit / etc/logrotate.d/haproxy.cfg, add the following:
/var/log/haproxy.log
{
rotate 4
daily
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
reload rsyslog > /dev/null 2>&1 || true
endscript
}
Startup script
$ wget -O haproxy https://raw.github.com/gist/3665034/4125bd5b81977a72e5eec30650fb21f3034782a0/haproxy-init.d $ cp haproxy /etc/init.d/haproxy $ chmod +x /etc/init.d/haproxy #Usage mode $ /etc/init.d/haproxy start|stop|restart
Install KeepAlived
Install dependency Libraries
$ yum install popt popt-devel
Install KeepAlived
$ wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz $ tar -zxvf keepalived-1.2.2.tar.gz $ cd keepalived-1.2.2 $ ./configure --prefix=/usr/local/keepalived $ make && make install $ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived $ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ $ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ $ mkdir -p /etc/keepalived/ $ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf $ chmod +x /etc/init.d/keepalived
Usage mode
$ /etc/init.d/keepalived start|stop|restart
Configuration on Master Server/etc/keepalived/keepalived.conf
global_defs {
notification_email {
user@example.com
}
notification_email_from mail@example.org
smtp_server 192.168.x.x
smtp_connect_timeout 30
router_id LVS_DEVEL
}
#Monitor haproxy process status and execute every 2 seconds
vrrp_script chk_haproxy {
script "/usr/local/keepalived/chk_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER #Marked as MASTER
interface eth0
virtual_router_id 51
priority 101 #MASTER weights are higher than BACKUP
advert_int 1
mcast_src_ip 192.168.1.189 #Master server IP
authentication {
auth_type PASS #Authentication Mode of Master-Slave Server
auth_pass 1111
}
track_script {
chk_haproxy #Monitoring haproxy process status
}
#VIP
virtual_ipaddress {
192.168.1.99 #Virtual IP
}
}
Configuration on Bakcup server/etc/keepalived/keepalived.conf
global_defs {
notification_email {
user@example.com
}
notification_email_from mail@example.org
smtp_server 192.168.x.x
smtp_connect_timeout 30
router_id LVS_DEVEL
}
#Monitor haproxy process status and execute every 2 seconds
vrrp_script chk_haproxy {
script "/usr/local/keepalived/chk_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state BACKUP #State BACKUP
interface eth0
virtual_router_id 51
priority 100 #Weight is lower than MASTER
advert_int 1
mcast_src_ip 192.168.1.222 #IP of Backup Server
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_haproxy #Monitoring haproxy process status
}
#VIP
virtual_ipaddress {
192.168.1.99 #Virtual IP
}
}
Content of chk_haproxy.sh
#!/bin/bash
#
# author: weizhifeng
# description:
# Check periodically whether haproxy exists or not, and if not, start haproxy.
# If startup fails, stop keeping alived
#
status=$(ps aux|grep haproxy | grep -v grep | grep -v bash | wc -l)
if [ "${status}" = "0" ]; then
/etc/init.d/haproxy start
status2=$(ps aux|grep haproxy | grep -v grep | grep -v bash |wc -l)
if [ "${status2}" = "0" ]; then
/etc/init.d/keepalived stop
fi
fi
High Availability Testing
-
Stop keeping alived on Master, check the system log, and find that MASTER releases VIP
$ /etc/init.d/keepalived stop $ tail -f /var/log/message Keepalived: Terminating on signal Keepalived: Stopping Keepalived v1.2.2 (11/03,2011) Keepalived_vrrp: Terminating VRRP child process on signal Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
-
Looking at the system log on Backup, it is found that Backup has entered the MASTER role and is bound to VIP 192.168.1.99.
$ tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.99 #Check on Backup to see if the VIP is already bound
-
Restart keepalived on Master, view the system log, find that the MASTER role has been regained, and bind VIP 192.168.1.99
$ /etc/init.d/keepalived start $ tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.99
-
Look at the system log on Backup and find that it has returned to the BACKUP role and released the VIP
$ tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Concurrent test
We use webbench to concurrently implement HAProxy test
$ yum install ctags $ wget http://home.tiscali.cz/~cz210552/distfiles/webbench-1.5.tar.gz $ tar -zxvf webbench-1.5.tar.gz $ cd webbench-1.5 $ make $ mkdir -p /usr/local/man && make install
Test environment:
CPU: Intel dual core x86_64 main frequency 3191MHZ
Mem: 2G
modify PHP - fpm.conf, set the number of processes for PHP-FPM span to 100:
pm.start_servers = 100 pm.max_spare_servers = 100
Test methods:
$ webbench -c 100 -t 3000 http://192.168.1.99/check.txt $ webbench -c 100 -t 3000 http://192.168.1.99/test.php
Test results:
Concurrent access to txt files, HAProxy session number is about 10,000, which shows that HAProxy can hold 10,000 concurrent connections; concurrent access to PHP files, HAProxy session peak is about 200, close to the concurrent processing capacity of back-end PHP (100x2).
Reference resources:
http://haproxy.1wt.eu/download/1.4/doc/configuration.txt http://kevin.vanzonneveld.net/techblog/article/haproxy_logging/