Returns and jobs of SaltStack_ Cache configuration and job management

Keywords: jobs

return of SaltStack component

The return component can be understood as the SaltStack system stores or returns the data returned by Minion to other programs. It supports a variety of storage methods, such as MySQL, MongoDB, Redis, Memcache, etc. through return, we can record each operation of SaltStack and provide a data source for future log audit. At present, 30 return data storage and interfaces are officially supported. We can easily configure and use it. Of course, it also supports self-defined returns. Custom returns need to be written in python. After selecting and configuring the return to use, just specify return after the salt command.

//View the list of all return s
[root@master ~]# salt 'p2' sys.list_returners
p2:
    - carbon
    - couchdb
    - etcd
    - highstate
    - local
    - local_cache
    - mattermost
    - multi_returner
    - pushover
    - rawfile_json
    - slack
    - slack_webhook
    - smtp
    - splunk
    - sqlite3
    - syslog
    - telegram

return process

Return is to trigger the task on the Master side, and then Minion accepts the processing task, directly establishes a connection with the return storage server, and then saves the data return to the storage server. It must be noted that the Minion side operates the storage server in this process, so it is necessary to ensure that the configuration and dependency package of the Minion side are correct, which means that we must install the specified return mode dependency package on each Minion. If Mysql is used as the return storage mode, we will install Python Mysql module on each Minion.

Use mysql as the return storage method

Environmental description

rolehost nameIP address
salt-mastermaster192.168.129.134
salt-minionp2192.168.129.250
mysql storage servermysql192.168.129.3

Note: returns and job cache cannot be started at the same time. Only one of them can be started

Install PyMySQL module on all minion s

[root@master ~]# salt 'p2' pkg.install python3-PyMySQL
p2:
    ----------
    python3-PyMySQL:
        ----------
        new:
            0.10.1-2.module_el8.4.0+666+456f5f48
        old:

[root@master ~]# salt 'p2' cmd.run 'rpm -qa |grep PyMySQL'
p2:
    python3-PyMySQL-0.10.1-2.module_el8.4.0+666+456f5f48.noarc

Deploy a mysql server as a storage server. Here, deploy it directly on the host 192.168.129.3
Add the data of three tables and select the data specified on the official website: https://docs.saltproject.io/en/latest/ref/returners/all/salt.returners.mysql.html

[root@mysql ~]# yum -y install mariadb mariadb-server
[root@mysql ~]# rpm -qa | grep mariadb
mariadb-5.5.68-1.el7.x86_64
mariadb-server-5.5.68-1.el7.x86_64
mariadb-libs-5.5.68-1.el7.x86_64

[root@mysql ~]# systemctl enable --now mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@mysql ~]# ss -anlt
State       Recv-Q Send-Q     Local Address:Port                    Peer Address:Port              
LISTEN      0      50                     *:3306                               *:*                  
LISTEN      0      128                    *:22                                 *:*                  
LISTEN      0      128                   :::22                                :::* 


//Create database and table structures
[root@mysql ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE  `salt`
    ->   DEFAULT CHARACTER SET utf8
    ->   DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]> 
MariaDB [(none)]> USE `salt`;
Database changed

MariaDB [salt]> CREATE TABLE `jids` (
    ->   `jid` varchar(255) NOT NULL,
    ->   `load` mediumtext NOT NULL,
    ->   UNIQUE KEY `jid` (`jid`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)

MariaDB [salt]> CREATE TABLE `salt_returns` (
    ->   `fun` varchar(50) NOT NULL,
    ->   `jid` varchar(255) NOT NULL,
    ->   `return` mediumtext NOT NULL,
    ->   `id` varchar(255) NOT NULL,
    ->   `success` varchar(10) NOT NULL,
    ->   `full_ret` mediumtext NOT NULL,
    ->   `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ->   KEY `id` (`id`),
    ->   KEY `jid` (`jid`),
    ->   KEY `fun` (`fun`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)

MariaDB [salt]> DROP TABLE IF EXISTS `salt_events`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [salt]> CREATE TABLE `salt_events` (
    -> `id` BIGINT NOT NULL AUTO_INCREMENT,
    -> `tag` varchar(255) NOT NULL,
    -> `data` mediumtext NOT NULL,
    -> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    -> `master_id` varchar(255) NOT NULL,
    -> PRIMARY KEY (`id`),
    -> KEY `tag` (`tag`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.00 sec)

MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids           |
| salt_events    |
| salt_returns   |
+----------------+
3 rows in set (0.00 sec)

//Authorized access
MariaDB [salt]> grant all on salt.* to salt@'%' identified by 'salt';
Query OK, 0 rows affected (0.001 sec)

MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.001 sec)

Test it to see if it can be stored

//Install mariadb on the minion side
[root@p2 ~]# yum -y install mariadb
[root@p2 ~]# which mysql
/usr/bin/mysql

[root@p2 ~]# mysql -usalt -psalt -h192.168.129.3
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> quit
Bye

Configure minion

[root@p2 ~]# vim /etc/salt/minion
.....Omitted here N that 's ok
mysql.host: '192.168.129.3'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306

[root@p2 ~]# systemctl restart salt-minion

Store the test in mysql on the Master

[root@master ~]# salt 'p2' test.ping 
p2:
    True
[root@master ~]# salt 'p2' test.ping --return mysql
p2:
    True
[root@master ~]# salt 'p2' cmd.run 'date' --return mysql
p2:
    Sun Nov  7 00:00:32 CST 2021

Query in database

MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
       fun: test.ping
       jid: 20211106155808772810
    return: true
        id: p2
   success: 1
  full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211106155808772810", "fun": "test.ping", "fun_args": [], "id": "p2"}
alter_time: 2021-11-06 23:58:08
*************************** 2. row ***************************
       fun: cmd.run
       jid: 20211106160032716714
    return: "Sun Nov  7 00:00:32 CST 2021"
        id: p2
   success: 1
  full_ret: {"success": true, "return": "Sun Nov  7 00:00:32 CST 2021", "retcode": 0, "jid": "20211106160032716714", "fun": "cmd.run", "fun_args": ["date"], "id": "p2"}
alter_time: 2021-11-07 00:00:32
2 rows in set (0.00 sec)

job cache

Note: job cache and returns cannot be started at the same time. Only one of them can be started

job cache process

When returning, the Minion directly interacts with the storage server. Therefore, it is necessary to install modules with specified storage methods on each Minion, such as Python mysql. Can we directly store the returned results on the Master to the storage server?

The answer is yes. This method is called job cache. It means that after Minion returns the results to the Master, the Master caches the results locally, and then stores the cached results to the specified storage server, such as mysql.

Operate according to the data of official documents: https://docs.saltproject.io/en/latest/topics/jobs/external_cache.html

Open the master on the master side_ job_ cache

[root@master ~]# yum -y install python3-PyMySQL
[root@master ~]# vim /etc/salt/master
....Omitted here N that 's ok
master_job_cache: mysql
mysql.host: '192.168.129.3'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306

[root@master ~]# systemctl restart salt-master

Test it to see if it can be stored

//Install mariadb on the master side
[root@master ~]# yum -y install mariadb
[root@master ~]# which mysql
/usr/bin/mysql

[root@master ~]# mysql -usalt -psalt -h192.168.129.3
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> quit
Bye

Empty table contents in database server

[root@mysql ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.3.28-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> delete from salt.salt_returns;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> select * from salt.salt_returns;
Empty set (0.000 sec)

Test again on the master to see if it can be stored in the database

[root@master ~]# salt 'p2' test.ping		#One ping does not work. Try several more times
p2:
    True
[root@master ~]# salt 'p2' cmd.run 'df -h'
p2:
    Filesystem             Size  Used Avail Use% Mounted on
    devtmpfs               883M     0  883M   0% /dev
    tmpfs                  900M   60K  900M   1% /dev/shm
    tmpfs                  900M  8.7M  892M   1% /run
    tmpfs                  900M     0  900M   0% /sys/fs/cgroup
    /dev/mapper/rhel-root   17G  1.8G   16G  11% /
    /dev/sda1             1014M  179M  836M  18% /boot
    tmpfs                  180M     0  180M   0% /run/user/0

Query in database

MariaDB [salt]> select * from salt.salt_returns\G
*************************** 1. row ***************************
       fun: test.ping
       jid: 20211106162452153906
    return: true
        id: p2
   success: 1
  full_ret: {"cmd": "_return", "id": "p2", "success": true, "return": true, "retcode": 0, "jid": "20211106162452153906", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-06T16:24:52.302145"}
alter_time: 2021-11-07 00:24:52
*************************** 2. row ***************************
       fun: cmd.run
       jid: 20211106162518269319
    return: "Filesystem             Size  Used Avail Use% Mounted on\ndevtmpfs               883M     0  883M   0% /dev\ntmpfs                  900M   60K  900M   1% /dev/shm\ntmpfs                  900M  8.7M  892M   1% /run\ntmpfs                  900M     0  900M   0% /sys/fs/cgroup\n/dev/mapper/rhel-root   17G  1.8G   16G  11% /\n/dev/sda1             1014M  179M  836M  18% /boot\ntmpfs                  180M     0  180M   0% /run/user/0"
        id: p2
   success: 1
  full_ret: {"cmd": "_return", "id": "p2", "success": true, "return": "Filesystem             Size  Used Avail Use% Mounted on\ndevtmpfs               883M     0  883M   0% /dev\ntmpfs                  900M   60K  900M   1% /dev/shm\ntmpfs                  900M  8.7M  892M   1% /run\ntmpfs                  900M     0  900M   0% /sys/fs/cgroup\n/dev/mapper/rhel-root   17G  1.8G   16G  11% /\n/dev/sda1             1014M  179M  836M  18% /boot\ntmpfs                  180M     0  180M   0% /run/user/0", "retcode": 0, "jid": "20211106162518269319", "fun": "cmd.run", "fun_args": ["df -h"], "_stamp": "2021-11-06T16:25:18.384847"}
alter_time: 2021-11-07 00:25:18
4 rows in set (0.00 sec)

job management

Gets the jid of the task

[root@master ~]# salt 'p2' cmd.run 'uptime' -v
Executing job with jid 20211106162903131351
-------------------------------------------

p2:
     00:29:03 up 50 min,  2 users,  load average: 0.03, 0.05, 0.06

Get the return result of this task through jid

[root@master ~]# salt-run jobs.lookup_jid 20211106162903131351
p2:
     00:29:03 up 50 min,  2 users,  load average: 0.03, 0.05, 0.06

[root@master ~]# salt-run jobs.list_jobs
20211106162255602000:
    ----------
    Arguments:
    Function:
        test.ping
    StartTime:
        2021, Nov 06 16:22:55.602000
    Target:
        *
    Target-type:
        glob
    User:
        root
20211106162300754793:
    ----------
    Arguments:
        - 20211106162255602000
    Function:
        saltutil.find_job
    StartTime:
        2021, Nov 06 16:23:00.754793
    Target:
        - p2
        - P1
    Target-type:
        list
    User:
        root
20211106162452153906:
    ----------
    Arguments:
    Function:
        test.ping
    StartTime:
        2021, Nov 06 16:24:52.153906
    Target:
        p2
    Target-type:
        glob
    User:
        root
20211106162518269319:
    ----------
    Arguments:
        - df -h
    Function:
        cmd.run
    StartTime:
        2021, Nov 06 16:25:18.269319
    Target:
        p2
    Target-type:
        glob
    User:
        root
20211106162818563006:
    ----------
    Arguments:
        - uptime
    Function:
        cmd.run
    StartTime:
        2021, Nov 06 16:28:18.563006
    Target:
        *
    Target-type:
        glob
    User:
        root
20211106162823706291:
    ----------
    Arguments:
        - 20211106162818563006
    Function:
        saltutil.find_job
    StartTime:
        2021, Nov 06 16:28:23.706291
    Target:
        - P1
    Target-type:
        list
    User:
        root
20211106162828237073:
    ----------
    Arguments:
        - uptime
    Function:
        cmd.run
    StartTime:
        2021, Nov 06 16:28:28.237073
    Target:
        p2
    Target-type:
        glob
    User:
        root
20211106162833282056:
    ----------
    Arguments:
        - 20211106162828237073
    Function:
        saltutil.find_job
    StartTime:
        2021, Nov 06 16:28:33.282056
    Target:
        - p2
    Target-type:
        list
    User:
        root
20211106162900461267:
    ----------
    Arguments:
    Function:
        test.ping
    StartTime:
        2021, Nov 06 16:29:00.461267
    Target:
        p2
    Target-type:
        glob
    User:
        root
20211106162903131351:
    ----------
    Arguments:
        - uptime
    Function:
        cmd.run
    StartTime:
        2021, Nov 06 16:29:03.131351
    Target:
        p2
    Target-type:
        glob
    User:
        root
20211106162937050489:
    ----------
    Arguments:
    Function:
        runner.jobs.lookup_jid
    StartTime:
        2021, Nov 06 16:29:37.050489
    Target:
        master_master
    Target-type:
        list
    User:
        root

[root@master ~]# salt-run jobs.active

Error message


Solution

//Install on the master side
[root@master ~]# yum -y install python3-PyMySQL

If the ping fails after installation, try several more times

Posted by doc on Sat, 06 Nov 2021 14:59:14 -0700

Hot Keywords

©2025 Programmer Group