Upgrade plan
Scope of influence:
During the upgrade, the existing system will not be affected, and the system will maintain normal operation. After the upgrade, the availability test will be conducted for a period of time, and the monitoring on production will be replaced after the system is stable.
Upgrade method:
This upgrade adopts the blue-green deployment method. First, redeploy a set of 4.0 in the test environment, import the original system configuration file into the test environment, and then upgrade from 4.0 to 5.0. Because 5.0 changes a lot compared with 4.0, and the number of tables and table fields in the database change a little compared with 4.0, installing 5.0 and importing 4.0 directly will lead to the failure of database data import.
Risks after upgrading:
Due to the short release time of official ZABBIX5.0, there may be some bugs in ZABBIX5.0. Later, the official will make some minor upgrades to fix the bugs
Upgrade principle:
- The newly built ZABBIX service and the original 4.0 environment should be the same, and the configuration of each service should be set according to the original 4.0 service configuration and ZABBIX monitoring content. If the minimum version required by some components does not meet the requirements, it should be upgraded. For example, if the memory usage of MySQL service in ZABBIX is too high, the memory configuration of MySQL service can be properly improved in the new environment.
- Do a good job of isolation between networks, and do not conflict with the original network environment
- Ensure that the original data is available and make a good backup. You may enter the original 4.0 environment to view some service configurations. Do not operate by mistake to cause data loss or service unavailability
- The host naming specification can understand the purpose of a single host through the host name. The ES cluster name should not conflict with the existing cluster name
- After the upgrade, the service availability requires that mysql master-slave synchronization is normal, the synchronization data of ES nodes is normal, the historical data collected by ES is normal, the health status of the cluster is green, the connection between zabbix proxy and server is normal, and the data can be obtained normally. The web service is built by lamp.
Upgrade ZABBIX
1. Close zabbix process and web process
systemctl stop zabbix-server systemctl stop zabbix-proxy systemctl stop httpd php-fpm
2. Backup 4.0 configuration data
mysqldump -uzabbix -p --single-transaction --master-data=2 --databases zabbix > olddata.sql # mkdir /opt/zabbix-backup/ # cp /etc/zabbix/zabbix_server.conf /opt/zabbix-backup/ # cp /etc/httpd/conf.d/zabbix.conf /opt/zabbix-backup/ # cp -R /usr/share/zabbix/ /opt/zabbix-backup/ # cp -R /usr/share/doc/zabbix-* /opt/zabbix-backup/
3. Update repository
Because the official source network is not good, I use Alibaba cloud's zabbix source here
rm -rf /etc/repos.d/zabbix.repo rpm -qa | grep zabbix-release && rpm -e zabbix-release rpm -Uvh https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm sed -i 's@repo.zabbix.com@mirrors.aliyun.com/zabbix@g' /etc/yum.repos.d/zabbix.repo yum clean all yum makecache fast
4. Upgrade zabbix
yum upgrade zabbix-server-mysql zabbix-web-mysql zabbix-agent
5. Upgrade ZABBIX web front end
yum remove zabbix-web yum install centos-release-scl -y yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-deps-scl-5.0.0-1.el7.noarch.rpm yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-5.0.0-1.el7.noarch.rpm yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-mysql-scl-5.0.0-1.el7.noarch.rpm yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-apache-conf-scl-5.0.0-1.el7.noarch.rpm
6. Modify php time zone
vim /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf php_value[date.timezone] = Asia/shanghai
7. Start zabbix
systemctl start zabbix-server zabbix-agent httpd rh-php72-php-fpm
8. This error may be encountered after startup
The official documents are not carefully read here, because the historical data and trend data (floating-point) data types of zabbix support higher precision and need to manually update the database patch,
The patch address of MySQL is: https://git.zabbix.com/projects/ZBX/repos/zabbix/raw/database/mysql/double.sql
ALTER TABLE trends MODIFY value_min DOUBLE PRECISION DEFAULT '0.0000' NOT NULL, MODIFY value_avg DOUBLE PRECISION DEFAULT '0.0000' NOT NULL, MODIFY value_max DOUBLE PRECISION DEFAULT '0.0000' NOT NULL; ALTER TABLE history MODIFY value DOUBLE PRECISION DEFAULT '0.0000' NOT NULL;
You also need to update the profile
vim /etc/zabbix/web/zabbix.conf.php $DB['DOUBLE_IEEE754'] = true;
Restart ZABBIX server after updating
systemctl restart zabbix-server
Upgrade Elasticsearch
zabbix5.0 now supports ES7.x version, and no longer supports the old version. One of the advantages of the upgrade is that ES supports index lifecycle management from version 6.6, which is more convenient for the historical data index management collected by zabbix. The newly upgraded zabbix of ES can directly deploy the 7.x es cluster. If you need the previous historical data, you can copy the original historical data from the original cluster. You can use the ES snapshot function or logstash to copy. The steps to upgrade es 6.1 to ES7.0 are described below. 6.1-7.0.1 does not support rolling upgrade, that is, the entire es cluster must be stopped for upgrade
Refer to the official document of ES: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/restart-upgrade.html
1. Close Zabbix docking ES
zabbix5.0 supports es of 7.x. before es is upgraded successfully, the writing of historical data to ES is temporarily stopped, and MySQL storage is used instead
Comment out part of docking ES
vim /etc/zabbix/web/zabbix.conf.php //$HISTORY['url'] = 'http://192.168.179.134:9200'; //$HISTORY['types']=['uint','dbl','str','log','text'];
vim /etc/zabbix/zabbix_server.conf #HistoryStorageURL=http://192.168.179.134:9200 #HistoryStorageTypes=uint,dbl,str,log,text #HistoryStorageDateIndex=1 systemctl restart zabbix-server
2. Disable replica sharding
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}3. Stop all nodes under ES cluster
systemctl stop elasticsearch
4. Upgrade all nodes
Since it was deployed in binary, I also use binary upgrade here
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.1-linux-x86_64.tar.gz tar -zxvf elasticsearch-7.0.1-linux-x86_64.tar.gz
Overwrite the configuration file of 6.1 to the config directory of 7.0.1
cp /elasticsearch/elasticsearch-6.4.0/config/elasticsearch.yml /elasticsearch-7.0.1/config/elasticsearch.yml
The modified configuration file is as follows, which needs to be defined in the cluster when starting the cluster for the first time cluster.initial_master_nodes, a group of nodes with master node qualification
5. Start all nodes
su elker ./elasticsearch-7.0.1/bin/elasticsearch
View the node status, and you can see that it has been upgraded to 7.0
Check the index and find that the previous index is still there 
6. Re enable sharding
When the primary partition recovery is completed and the cluster status is yellow, the partition allocation can be re enabled
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}7. Modify the configuration file to connect zabbix to ES
vim /etc/zabbix/web/zabbix.conf.php $HISTORY['url'] = 'http://192.168.179.134:9200'; $HISTORY['types']=['uint','dbl','str','log','text'];
vim /etc/zabbix/zabbix_server.conf HistoryStorageURL=http://192.168.179.134:9200 HistoryStorageTypes=uint,dbl,str,log,text HistoryStorageDateIndex=1 systemctl restart zabbix-server
Welcome to pay attention to the "operation and maintenance development story" of personal company