ZABBIX4.0 upgrade 5.0 & ES6.1 upgrade 7.0

Keywords: Operation & Maintenance Zabbix ElasticSearch yum MySQL

;

Upgrade plan

Scope of influence:

During the upgrade, the existing system will not be affected, and the system will maintain normal operation. After the upgrade, the availability test will be conducted for a period of time, and the monitoring on production will be replaced after the system is stable.

Upgrade method:

This upgrade adopts the blue-green deployment method. First, redeploy a set of 4.0 in the test environment, import the original system configuration file into the test environment, and then upgrade from 4.0 to 5.0. Because 5.0 changes a lot compared with 4.0, and the number of tables and table fields in the database change a little compared with 4.0, installing 5.0 and importing 4.0 directly will lead to the failure of database data import.

Risks after upgrading:

Due to the short release time of official ZABBIX5.0, there may be some bugs in ZABBIX5.0. Later, the official will make some minor upgrades to fix the bugs

Upgrade principle:

  • The newly built ZABBIX service and the original 4.0 environment should be the same, and the configuration of each service should be set according to the original 4.0 service configuration and ZABBIX monitoring content. If the minimum version required by some components does not meet the requirements, it should be upgraded. For example, if the memory usage of MySQL service in ZABBIX is too high, the memory configuration of MySQL service can be properly improved in the new environment.
  • Do a good job of isolation between networks, and do not conflict with the original network environment
  • Ensure that the original data is available and make a good backup. You may enter the original 4.0 environment to view some service configurations. Do not operate by mistake to cause data loss or service unavailability
  • The host naming specification can understand the purpose of a single host through the host name. The ES cluster name should not conflict with the existing cluster name
  • After the upgrade, the service availability requires that mysql master-slave synchronization is normal, the synchronization data of ES nodes is normal, the historical data collected by ES is normal, the health status of the cluster is green, the connection between zabbix proxy and server is normal, and the data can be obtained normally. The web service is built by lamp.

Upgrade ZABBIX

1. Close zabbix process and web process

systemctl stop zabbix-server
systemctl stop zabbix-proxy
systemctl stop httpd php-fpm

2. Backup 4.0 configuration data

mysqldump -uzabbix -p --single-transaction --master-data=2 --databases zabbix > olddata.sql
# mkdir /opt/zabbix-backup/
# cp /etc/zabbix/zabbix_server.conf /opt/zabbix-backup/
# cp /etc/httpd/conf.d/zabbix.conf  /opt/zabbix-backup/
# cp -R /usr/share/zabbix/ /opt/zabbix-backup/
# cp -R /usr/share/doc/zabbix-* /opt/zabbix-backup/

3. Update repository

Because the official source network is not good, I use Alibaba cloud's zabbix source here

rm -rf /etc/repos.d/zabbix.repo
rpm -qa | grep zabbix-release && rpm -e zabbix-release
rpm -Uvh https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
sed -i 's@repo.zabbix.com@mirrors.aliyun.com/zabbix@g' /etc/yum.repos.d/zabbix.repo
yum clean all
yum makecache fast

4. Upgrade zabbix

 yum upgrade zabbix-server-mysql zabbix-web-mysql zabbix-agent

5. Upgrade ZABBIX web front end

yum remove zabbix-web
yum install centos-release-scl -y
yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-deps-scl-5.0.0-1.el7.noarch.rpm
yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-5.0.0-1.el7.noarch.rpm
yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-web-mysql-scl-5.0.0-1.el7.noarch.rpm
yum install https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/frontend/zabbix-apache-conf-scl-5.0.0-1.el7.noarch.rpm

6. Modify php time zone

vim /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
php_value[date.timezone] = Asia/shanghai

7. Start zabbix

systemctl  start zabbix-server zabbix-agent httpd rh-php72-php-fpm

8. This error may be encountered after startup

The official documents are not carefully read here, because the historical data and trend data (floating-point) data types of zabbix support higher precision and need to manually update the database patch,
The patch address of MySQL is: https://git.zabbix.com/projects/ZBX/repos/zabbix/raw/database/mysql/double.sql

ALTER TABLE trends MODIFY value_min DOUBLE PRECISION DEFAULT '0.0000' NOT NULL, MODIFY value_avg DOUBLE PRECISION DEFAULT '0.0000' NOT NULL, MODIFY value_max DOUBLE PRECISION DEFAULT '0.0000' NOT NULL;
ALTER TABLE history MODIFY value DOUBLE PRECISION DEFAULT '0.0000' NOT NULL;

You also need to update the profile

vim /etc/zabbix/web/zabbix.conf.php
$DB['DOUBLE_IEEE754'] = true;

Restart ZABBIX server after updating

systemctl restart zabbix-server

Upgrade Elasticsearch

zabbix5.0 now supports ES7.x version, and no longer supports the old version. One of the advantages of the upgrade is that ES supports index lifecycle management from version 6.6, which is more convenient for the historical data index management collected by zabbix. The newly upgraded zabbix of ES can directly deploy the 7.x es cluster. If you need the previous historical data, you can copy the original historical data from the original cluster. You can use the ES snapshot function or logstash to copy. The steps to upgrade es 6.1 to ES7.0 are described below. 6.1-7.0.1 does not support rolling upgrade, that is, the entire es cluster must be stopped for upgrade
Refer to the official document of ES: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/restart-upgrade.html

1. Close Zabbix docking ES

zabbix5.0 supports es of 7.x. before es is upgraded successfully, the writing of historical data to ES is temporarily stopped, and MySQL storage is used instead
Comment out part of docking ES

vim /etc/zabbix/web/zabbix.conf.php
//$HISTORY['url'] = 'http://192.168.179.134:9200';
//$HISTORY['types']=['uint','dbl','str','log','text'];
vim /etc/zabbix/zabbix_server.conf
#HistoryStorageURL=http://192.168.179.134:9200
#HistoryStorageTypes=uint,dbl,str,log,text
#HistoryStorageDateIndex=1
systemctl restart zabbix-server

2. Disable replica sharding

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

3. Stop all nodes under ES cluster

systemctl stop elasticsearch

4. Upgrade all nodes

Since it was deployed in binary, I also use binary upgrade here

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.1-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.0.1-linux-x86_64.tar.gz

Overwrite the configuration file of 6.1 to the config directory of 7.0.1

cp /elasticsearch/elasticsearch-6.4.0/config/elasticsearch.yml /elasticsearch-7.0.1/config/elasticsearch.yml

The modified configuration file is as follows, which needs to be defined in the cluster when starting the cluster for the first time cluster.initial_master_nodes, a group of nodes with master node qualification

5. Start all nodes

su elker
./elasticsearch-7.0.1/bin/elasticsearch

View the node status, and you can see that it has been upgraded to 7.0

Check the index and find that the previous index is still there

6. Re enable sharding

When the primary partition recovery is completed and the cluster status is yellow, the partition allocation can be re enabled

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

7. Modify the configuration file to connect zabbix to ES

vim /etc/zabbix/web/zabbix.conf.php
$HISTORY['url'] = 'http://192.168.179.134:9200';
$HISTORY['types']=['uint','dbl','str','log','text'];
vim /etc/zabbix/zabbix_server.conf
HistoryStorageURL=http://192.168.179.134:9200
HistoryStorageTypes=uint,dbl,str,log,text
HistoryStorageDateIndex=1
systemctl restart zabbix-server

Welcome to pay attention to the "operation and maintenance development story" of personal company

Posted by koolswans on Sun, 31 May 2020 10:01:07 -0700