return and job management of SaltStack
return of SaltStack component
The return component can be understood as the SaltStack system stores or returns the data returned by Minion to other programs. It supports a variety of storage methods, such as MySQL, MongoDB, Redis, Memcache, etc. through return, we can record each operation of SaltStack and provide a data source for future log audit. At present, 30 return data storage and interfaces are officially supported. We can easily configure and use it. Of course, it also supports self-defined returns. Custom returns need to be written in python. After selecting and configuring the return to use, just specify return after the salt command.
# View the list of all return s on node1
[root@master ~]# salt node1 sys.list_returners
node1:
- carbon
- couchdb
- etcd
- highstate
- local
- local_cache
- mattermost
- multi_returner
- pushover
- rawfile_json
- slack
- slack_webhook
- smtp
- splunk
- sqlite3
- syslog
- telegram
return process
Return is to trigger the task on the Master side, and then Minion accepts the processing task, directly establishes a connection with the return storage server, and then saves the data return to the storage server. It must be noted that the Minion side operates the storage server in this process, so it is necessary to ensure that the configuration and dependency package of the Minion side are correct, which means that we must install the specified return mode dependency package on each Minion. If Mysql is used as the return storage mode, we will install Python Mysql module on each Minion.
Use mysql as the return storage method
Environmental description:
| host | IP | service |
|---|---|---|
| master | 192.168.10.201 | salt-master |
| minion(node1) | 192.168.10.202 | salt-minion mariadb |
| mariadb(node2) | 192.168.10.203 | mariadb-server mariadb |
Start service on master
[root@master ~]# systemctl start salt-master.service [root@master ~]# systemctl stop salt-minion.service [root@master ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 0.0.0.0:4505 0.0.0.0:* LISTEN 0 128 0.0.0.0:4506 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:*
Install python3 pymysqlbing startup service on node1
[root@node1 ~]# systemctl start salt-minion.service
[root@node1 ~]# yum -y install python3-PyMySQL
[root@node1 ~]# systemctl status salt-minion.service
● salt-minion.service - The Salt Minion
Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; ena>
Active: active (running) since Sat 2021-11-06 23:26:10 EDT; 1h 3>
Docs: man:salt-minion(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltproject.io/en/latest/contents.html
Main PID: 967 (salt-minion)
Tasks: 6 (limit: 11201)
Memory: 120.1M
CGroup: /system.slice/salt-minion.service
├─ 967 /usr/bin/python3.6 /usr/bin/salt-minion
├─1430 /usr/bin/python3.6 /usr/bin/salt-minion
└─1438 /usr/bin/python3.6 /usr/bin/salt-minion
11 June 23:26:03 node1 systemd[1]: Starting The Salt Minion...
11 June 23:26:10 node1 systemd[1]: Started The Salt Minion.
mariadb installs the service and configures it
[root@node2 ~]# yum -y install mariadb mariadb-serve
[root@node2 ~]# systemctl start mariadb.service
[root@node2 ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 80 *:3306 *:*
LISTEN 0 32 *:21 *:*
LISTEN 0 128 [::]:22 [::]:*
[root@node2 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.28-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> set password = password("lq123!");
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> CREATE DATABASE `salt` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.001 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| salt |
+--------------------+
4 rows in set (0.001 sec)
MariaDB [(none)]> use salt;
Database changed
MariaDB [salt]> DROP TABLE IF EXISTS `jids`;
Query OK, 0 rows affected, 1 warning (0.001 sec)
MariaDB [salt]> CREATE TABLE `jids` (`jid` varchar(255) NOT NULL,`load` mediumtext NOT NULL, UNIQUE KEY `jid` (`jid`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.015 sec)
MariaDB [salt]> DROP TABLE IF EXISTS `salt_returns`;
Query OK, 0 rows affected, 1 warning (0.000 sec)
MariaDB [salt]> CREATE TABLE `salt_returns` (
-> `fun` varchar(50) NOT NULL,
-> `jid` varchar(255) NOT NULL,
-> `return` mediumtext NOT NULL,
-> `id` varchar(255) NOT NULL,
-> `success` varchar(10) NOT NULL,
-> `full_ret` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> KEY `id` (`id`),
-> KEY `jid` (`jid`),
-> KEY `fun` (`fun`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.020 sec)
MariaDB [salt]> CREATE TABLE `salt_events` (
-> `id` BIGINT NOT NULL AUTO_INCREMENT,
-> `tag` varchar(255) NOT NULL,
-> `data` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> `master_id` varchar(255) NOT NULL,
-> PRIMARY KEY (`id`),
-> KEY `tag` (`tag`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.003 sec)
##Authorized access
MariaDB [salt]> grant all on salt.* to salt@'%' identified by 'salt';
Query OK, 0 rows affected (0.001 sec)
MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.000 sec)
Receive node1 authentication on the master
[root@master ~]# salt-key -L
Accepted Keys:
master
node1
node2
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[root@master ~]# salt '*' test.ping
node1:
True
node2:
True
master:
True
Installing mariadb on node1
[root@node1 ~]# yum -y install mariadb ##Log in using the user and password created by the mariadb host [root@node1 ~]# mysql -usalt -psalt -h192.168.240.40 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> exit Bye ##Modify profile [root@node1 ~]# vim /etc/salt/minion #return: # - mysql # - hipchat # - slack mysql.host: '192.168.240.40' mysql.user: 'salt' mysql.pass: 'salt' mysql.db: 'salt' mysql.port: 3306 ###### Miscellaneous settings ###### ##Restart minion [root@node1 ~]# systemctl restart salt-minion.service
ping the master to test connectivity
[root@master ~]# salt '*' test.ping
node1:
True
node2:
True
master:
True
Check for data on mariadb
[root@node2 ~]# mysql -uroot -plq123! Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 10 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> select * from salt.salt_returns; Empty set (0.003 sec)
The tests on the master are stored in mariadb
[root@master ~]# salt 'node1' test.ping --return mysql
node1:
True
mariadb view
MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211107055302740262
return: true
id: node1
success: 1
full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211107055302740262", "fun": "test.ping", "fun_args": [], "id": "node1"}
alter_time: 2021-11-07 01:53:02
1 row in set (0.002 sec)
job cache
job cache process
When returning, the Minion directly interacts with the storage server. Therefore, it is necessary to install modules with specified storage methods on each Minion, such as Python mysql. Can we directly store the returned results on the Master to the storage server?
The answer is yes. This method is called job cache. It means that after Minion returns the results to the Master, the Master caches the results locally, and then stores the cached results to the specified storage server, such as mysql.
Sketch Map
master default storage location
[root@master ~]# cd /var/cache/salt/master/jobs/ [root@master jobs]# ls 00 0e 22 2e 3a 47 53 65 70 7e 8d aa b3 c5 d1 e9 01 14 23 2f 3b 49 54 66 72 7f 8e ab b4 c7 d7 ea 02 16 25 30 3c 4a 56 68 73 80 93 ac b5 c9 dd ef 03 17 26 32 3d 4c 57 69 74 84 94 ad b6 ca e0 f0 05 18 27 33 3f 4d 58 6a 76 88 98 ae b7 cc e1 f2 06 19 29 35 40 50 5c 6b 77 89 99 af b9 cd e6 f3 08 1b 2b 36 44 51 5d 6c 78 8a 9b b0 c1 ce e7 f7 0d 1f 2d 39 46 52 5f 6e 79 8c a8 b1 c2 cf e8 f9
Comment out the configuration file on node1 that was changed before
[root@node1 ~]# vim /etc/salt/minion # - hipchat # - slack #mysql.host: '192.168.240.40' #mysql.user: 'salt' #mysql.pass: 'salt' #mysql.db: 'salt' #mysql.port: 3306 ###### Miscellaneous settings ###### ############################################ [root@node1 ~]# systemctl restart salt-minion.service
Open the master on the master side_ job_ cache
[root@master jobs]# yum -y install python3-PyMySQL ##Modify the master configuration file [root@master jobs]# vim /etc/salt/master 137 #job_cache: True 138 mysql.host: '192.168.220.40' 139 mysql.user: 'salt' 140 mysql.pass: 'salt' 141 mysql.db: 'salt' 142 mysql.port: 3306 143 # Cache minion grains, pillar and mine data via the cache subsy stem in the ##Restart the master [root@master jobs]# systemctl restart salt-master.service
Test connectivity
[root@master jobs]# salt 'node1' test.ping
node1:
True
master installation mariadb service
[root@master jobs]# yum -y install mariadb ##Login with salt user password [root@master jobs]# mysql -usalt -psalt -h192.168.240.40 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 15 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
Delete previous data on mariadb
[root@node2 ~]# mysql -uroot -plq123! Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 12 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> delete from salt.salt_returns; Query OK, 1 row affected (0.003 sec) MariaDB [(none)]> select * from salt.salt_returns; Empty set (0.000 sec) MariaDB [(none)]> exit Bye
Test again on the master to see if it can be stored in the database
[root@master jobs]# salt 'node1' test.ping --return mysql
node1:
True
Query in mariadb database
MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211107063529128883
return: true
id: node1
success: 1
full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211107063529128883", "fun": "test.ping", "fun_args": [], "id": "node1"}
alter_time: 2021-11-07 01:35:29
1 row in set (0.000 sec)
job management
Salt 0.9.7 is the of management jobs saltutil Some new features have been introduced. These functions are:
- Running returns the data of all running jobs found in the proc directory.
- find_ The job returns specific data about a job according to the job ID.
- signal_job allows sending signals to a given fixture.
- term_ The job sends a termination signal to the process controlling the specified job (SIGTERM, 15).
- kill_ The job sends a kill signal to the process controlling the specified job (SIGKILL, 9).
Gets the jid of the task
[root@master ~]# salt '*' cmd.run 'date' -v
Executing job with jid 20211107063705962754
-------------------------------------------
node1:
Sun Nov 7 01:37:06 EST 2021
node2:
Sun Nov 7 01:37:06 EST 2021
master:
Sun Nov 7 01:37:06 EST 2021
Get the return result of the task through jid
[root@master ~]# salt-run jobs.lookup_jid 20211107063705962754
master:
Sun Nov 7 01:37:06 EST 2021
node1:
Sun Nov 7 01:37:06 EST 2021
node2:
Sun Nov 7 01:37:06 EST 2021
List the tasks being executed. You can use kill above_ Job Jid kills an executing task
[root@master ~]# salt-run jobs.active
List tasks performed
[root@master ~]# salt-run jobs.list_jobs
20211107033832842417:
----------
Arguments:
Function:
test.ping
StartTime:
2021, Nov 07 03:38:32.842417
Target:
*
Target-type:
glob
User:
root
20211107034017336895:
----------
Arguments:
Function:
sys.list_returners
StartTime:
2021, Nov 07 03:40:17.336895
Target:
node1
Target-type:
glob
User:
root