return and job management of SaltStack

Experimental environment

1. return of saltstack component

The return component can be understood as the SaltStack system stores or returns the data returned by Minion to other programs. It supports a variety of storage methods, such as MySQL, MongoDB, Redis, Memcache, etc. through return, we can record each operation of SaltStack and provide a data source for future log audit. At present, 30 return data storage and interfaces are officially supported. We can easily configure and use it. Of course, it also supports self-defined returns. Custom returns need to be written in python. After selecting and configuring the return to use, just specify return after the salt command.

//View the list of all return s
[root@master ~]# salt '*' sys.list_returners
master:
    - carbon
    - couchdb
    - etcd
    - highstate
    - local
    - local_cache
    - mattermost
    - multi_returner
    - pushover
    - rawfile_json
    - slack
    - slack_webhook
    - smtp
    - splunk
    - sqlite3
    - syslog
    - telegram
192.168.200.147:
    - carbon
    - couchdb
    - etcd
    - highstate
    - local
    - local_cache
    - mattermost
    - multi_returner
    - pushover
    - rawfile_json
    - slack
    - slack_webhook
    - smtp
    - splunk
    - sqlite3
    - syslog
    - telegram

1.1 return process

Return is to trigger the task on the Master side, and then Minion accepts the processing task, directly establishes a connection with the return storage server, and then saves the data return to the storage server. It must be noted that the Minion side operates the storage server in this process, so it is necessary to ensure that the configuration and dependency package of the Minion side are correct, which means that we must install the specified return mode dependency package on each Minion. If Mysql is used as the return storage mode, we will install the python Mysql module on each Minion

1.2 use mysql as the return storage method

//Install mariadb on both the server and client
[root@master ~]# yum -y install mariadb-server mariadb
[root@node1 ~]# yum -y install mariadb-server mariadb
[root@node2 ~]# yum -y install mariadb-server mariadb

//Set startup and self startup
[root@master ~]# systemctl enable --now mariadb
[root@node1 ~]# systemctl enable --now  mariadb 
[root@node2 ~]# systemctl enable --now  mariadb

//Install pymysql
[root@master ~]# dnf -y install python3-PyMySQL
[root@node1 ~]# dnf -y install  python3-PyMySQL
[root@node2 ~]# dnf -y install  python3-PyMySQL

//Create a database and table structure
MariaDB [(none)]> CREATE DATABASE  `salt`
    ->   DEFAULT CHARACTER SET utf8
    ->   DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.001 sec)

MariaDB [(none)]> USE `salt`;
Database changed
MariaDB [salt]> DROP TABLE IF EXISTS `jids`;
Query OK, 0 rows affected, 1 warning (0.011 sec)

MariaDB [salt]> CREATE TABLE `jids` (
    ->   `jid` varchar(255) NOT NULL,
    ->   `load` mediumtext NOT NULL,
    ->   UNIQUE KEY `jid` (`jid`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.005 sec)

MariaDB [salt]> CREATE TABLE `salt_returns` (
    ->   `fun` varchar(50) NOT NULL,
    ->   `jid` varchar(255) NOT NULL,
    ->   `return` mediumtext NOT NULL,
    ->   `id` varchar(255) NOT NULL,
    ->   `success` varchar(10) NOT NULL,
    ->   `full_ret` mediumtext NOT NULL,
    ->   `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ->   KEY `id` (`id`),
    ->   KEY `jid` (`jid`),
    ->   KEY `fun` (`fun`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.012 sec)

MariaDB [salt]> CREATE TABLE `salt_events` (
    -> `id` BIGINT NOT NULL AUTO_INCREMENT,
    -> `tag` varchar(255) NOT NULL,
    -> `data` mediumtext NOT NULL,
    -> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    -> `master_id` varchar(255) NOT NULL,
    -> PRIMARY KEY (`id`),
    -> KEY `tag` (`tag`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.002 sec)

MariaDB [salt]>  grant all on salt.* to salt@'%' identified by 'salt';
Query OK, 0 rows affected (0.001 sec)

MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.000 sec)

//connect
[root@node2 ~]# mysql -usalt -psalt -h192.168.200.147
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.28-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> 


//configuration file
[root@node1 ~]# vim /etc/salt/minion
.....
mysql.host: '192.168.200.147'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
[root@node1 ~]# systemctl restart salt-minion.service 

//test
[root@master ~]# salt '*' test.ping --return mysql 
master:
    True
192.168.200.145:
    True
192.168.200.147:
    True
    
MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
       fun: test.ping
       jid: 20211107040133598049
    return: true
        id: 192.168.200.147
   success: 1
  full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211107040133598049", "fun": "test.ping", "fun_args": [], "id": "192.168.200.147"}
alter_time: 2021-11-07 00:01:49
1 row in set (0.000 sec)

2. job cache

2.1 job cache process

When returning, the Minion directly interacts with the storage server. Therefore, it is necessary to install modules with specified storage methods on each Minion, such as Python mysql. Can we directly store the returned results on the Master to the storage server?

The answer is yes. This method is called job cache. It means that after Minion returns the results to the Master, the Master caches the results locally, and then stores the cached results to the specified storage server, such as mysql

Open the master on the master side_ job_ cache

[root@master ~]# vim /etc/salt/master
.....
#job_cache: True
master_job_cache: mysql
mysql.host: '192.168.200.147'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
[root@master ~]# systemctl restart salt-master.service

Empty table contents in database server

MariaDB [(none)]> delete from salt.salt_returns;
Query OK, 7 rows affected (0.001 sec)

MariaDB [(none)]> select * from salt.salt_returns;
Empty set (0.000 sec)

Test again on the master to see if it can be stored in the database

[root@master ~]# salt '*' cmd.run 'df -h'
master:
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               3.8G     0  3.8G   0% /dev
    tmpfs                  3.9G  360K  3.9G   1% /dev/shm
    tmpfs                  3.9G   17M  3.8G   1% /run
    tmpfs                  3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/mapper/rhel-root   44G  3.0G   42G   7% /
    /dev/nvme0n1p1        1014M  179M  836M  18% /boot
    tmpfs                  779M     0  779M   0% /run/user/0
192.168.200.145:
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               883M     0  883M   0% /dev
    tmpfs                  900M   40K  900M   1% /dev/shm
    tmpfs                  900M  8.7M  892M   1% /run
    tmpfs                  900M     0  900M   0% /sys/fs/cgroup
    /dev/mapper/rhel-root   17G  2.0G   16G  12% /
    /dev/nvme0n1p1        1014M  179M  836M  18% /boot
    tmpfs                  180M     0  180M   0% /run/user/0
192.168.200.147:
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               883M     0  883M   0% /dev
    tmpfs                  901M  120K  901M   1% /dev/shm
    tmpfs                  901M  8.8M  892M   1% /run
    tmpfs                  901M     0  901M   0% /sys/fs/cgroup
    /dev/mapper/rhel-root   44G  2.4G   42G   6% /
    /dev/nvme0n1p1        1014M  179M  836M  18% /boot
    tmpfs                  181M     0  181M   0% /run/user/0

MariaDB [(none)]>  select * from salt.salt_returns\G
Empty set (0.000 sec)

*************************** 7. row ***************************
       fun: cmd.run
       jid: 20211107051330647194
    return: "Filesystem             Size  Used Avail Use% Mounted on\ndevtmpfs               883M     0  883M   0% /dev\ntmpfs                  900M   40K  900M   1% /dev/shm\ntmpfs                  900M  8.7M  892M   1% /run\ntmpfs                  900M     0  900M   0% /sys/fs/cgroup\n/dev/mapper/rhel-root   17G  2.0G   16G  12% /\n/dev/nvme0n1p1        1014M  179M  836M  18% /boot\ntmpfs                  180M     0  180M   0% /run/user/0"
        id: 192.168.200.145
   success: 1
  full_ret: {"cmd": "_return", "id": "192.168.200.145", "success": true, "return": "Filesystem             Size  Used Avail Use% Mounted on\ndevtmpfs               883M     0  883M   0% /dev\ntmpfs                  900M   40K  900M   1% /dev/shm\ntmpfs                  900M  8.7M  892M   1% /run\ntmpfs                  900M     0  900M   0% /sys/fs/cgroup\n/dev/mapper/rhel-root   17G  2.0G   16G  12% /\n/dev/nvme0n1p1        1014M  179M  836M  18% /boot\ntmpfs                  180M     0  180M   0% /run/user/0", "retcode": 0, "jid": "20211107051330647194", "fun": "cmd.run", "fun_args": ["df -h"], "_stamp": "2021-11-07T05:13:30.847203"}
alter_time: 2021-11-07 01:13:30

2.2 job management

Gets the jid of the task

[root@master ~]# salt '*' cmd.run 'uptime' -v
Executing job with jid 20211107052528839558
-------------------------------------------

master:
     01:25:29 up  3:53,  3 users,  load average: 0.01, 0.06, 0.07
192.168.200.145:
     13:25:29 up  2:13,  2 users,  load average: 0.07, 0.10, 0.04
192.168.200.147:
     01:25:29 up  2:13,  3 users,  load average: 0.23, 0.30, 0.26

Get the return result of this task through jid

[root@master ~]# salt '*' cmd.run 'uptime' -v
Executing job with jid 20211107052528839558
-------------------------------------------

master:
     01:32:47 up  4:01,  3 users,  load average: 0.03, 0.08, 0.08
192.168.200.145:
     13:32:48 up  2:20,  2 users,  load average: 0.01, 0.05, 0.03
192.168.200.147:
     01:32:48 up  2:21,  3 users,  load average: 0.02, 0.11, 0.17

[root@master ~]# salt-run jobs.lookup_jid 20211107052528839558
192.168.200.145:
     13:25:29 up  2:13,  2 users,  load average: 0.07, 0.10, 0.04
192.168.200.147:
     01:25:29 up  2:13,  3 users,  load average: 0.23, 0.30, 0.26
master:
     01:25:29 up  3:53,  3 users,  load average: 0.01, 0.06, 0.07