After investigation, it was found that the disk space of the local test environment was full, which made the database unable to write data.
The ultimate culprit is log files.
Because we are using lvs load balancing, the front-end Keepalived can not initialize the ipvs protocol after starting, so the content of the log is constantly brushing, the speed is amazing, the size of the log file is constantly growing crazily, resulting in insufficient space, cup things happened.
Wrong content
View System Log File Content
tail -f /var/log/messages
You'll see a lot of the following information beating.
May 16 08:35:22 linux-node1 Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.144 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.146 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink reflector May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink command channel May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'. May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Configuration is using : 13073 Bytes May 16 08:35:22 linux-node1 Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.144 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Netlink reflector reports IP 10.20.23.146 added May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink reflector May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Registering Kernel netlink command channel May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'. May 16 08:35:22 linux-node1 Keepalived_healthcheckers: Configuration is using : 13073 Bytes
Look at the phrase "IPVS: Can't initialize ipvs: Protocol not available". It's already obvious that the IPVS protocol can't be initialized. Look at the solution.
Solution
First you have to install ipvsadm
yum install ipvsadm
If you've already installed it, just ignore it and execute the command afterwards.
ipvsadm
This little command will help you solve the problem above. Now you can see if your message has been refreshed all the time. Haha.~~~
[root@linux-node1 keepalived]# ipvsadm IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.20.23.146:http wrr persistent 300 -> web02:0 Route 1 0 0 -> 10.20.23.250:0 Route 1 0 0