saltstack efficient operation and maintenance

Keywords: Linux Nginx Python saltstack CentOS

saltstack efficient operation and maintenance


salt introduction

saltstack is an open source project created by thomas Hatch in 2011. It is designed to implement a fast remote execution system.

Is salt powerful

System administrators do a lot of repetitive operations, such as installing software, modifying configuration files, creating users, executing commands in batches, etc. If the number of hosts is huge, it's really unbearable to rely on manual maintenance alone.

Early operation and maintenance personnel will write specific scripts according to their own production environment to complete a lot of repetitive work. These scripts are complex and difficult to maintain. System administrators are mainly faced with the following problems: 1. System configuration management; 2. Remote command execution. As a result, many open-source software has been born. In terms of system maintenance, there are fabric, puppet, chef, ansible, saltstack, etc. these software are good at maintaining system status or performing batch command execution on a large number of hosts conveniently.

Salt is flexible and can be deployed on a large scale or on a small scale. The design architecture of salt is suitable for any number of servers. From a small number of local network systems to spanning several data centers, the topology architecture is a c/s model with simple configuration.

No matter how many, hundreds or thousands of servers, you can use salt to control on a central node, and flexibly locate any subset of servers to run commands.  

Salt is written by python. It supports users to customize functional modules through python, and also provides a large number of python API interfaces. Users can expand it simply and quickly according to their needs.

How saltstack works

  • Local operation, delivery management
  • Master/Minion
  • Salt SSH does not require a client

salt deployment infrastructure

Before installing salt, understand the roles in the salt architecture. The main differences are salt master and salt minion. As the name implies, master is the central control system and minion is the managed client.

One of the salt architectures is master > minion.

In the remote execution system, salt uses python to complete the task through function calls.

Run salt dependency package


The easiest way to solve the dependency package is to use the installation package manager, yum tool

Server environment preparation

Server environment centos7(master) centos7(master)

ip address


master slave

software package

salt-master salt-minion

Change the ip address of virtual machine to static address, and make sure it can access the Internet

master network card configuration
slave network card configuration
# Generated by NetworkManager
dns profile   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6  slave  master

Turn off server security policy

Turn off firewalld
systemctl disable firewalld
systemctl stop firewalld

Close iptables
iptables -F

Turn off selinux

Install saltstack

The salt package needs the support of epel source, then download

The full name of EPEL is Extra Packages for Enterprise Linux. EPEL is a project built by Fedora community to provide high-quality software packages for RHEL and derivative distributions such as CentOS and Scientific Linux. After you install EPEL, you add a third-party source.

wget -O /etc/yum.repos.d/CentOS-Base.repo
wget -O /etc/yum.repos.d/epel.repo
Clear all? Clear cache
yum makecache? Generates the yum cache

After installing the epel source, you can start installing salt

View salt packages
yum list salt

Install salt master yum install salt-master -y
install salt-minion
yum install salt-minion -y

salt port

Start configuration after installing salt. The salt master listens to two ports by default:

4505 publish? Port provides remote command sending function
 4506 ret_port provides authentication, file service, result collection and other functions
 Ensure that the client can communicate with the two ports of the server, and ensure that the firewall allows the ports to pass through. So shut down the firewall directly in the test environment.

configuration file

The configuration file of salt master is / etc/salt/master
 The configuration file for salt minion is / etc/salt/minion
 The configuration file contains a large number of adjustable parameters, which control all aspects of the master and minion

Configure salt master

# salt Users running, affecting salt Execution permission of
user: root

#s alt The more threads are opened, the faster the processing speed is, but generally, it should not exceed CPU Number of
worker_threads: 10

# master Management port for
publish_port : 4505

# master Follow minion Communication port for file service, authentication, acceptance of returned results, etc
ret_port : 4506

# If this master Running salt-syndic Connected to a higher level master,Then this parameter needs to be configured to connect to the higher level master Listening port for
syndic_master_port : 4506

# Appoint pid file location
pidfile: /var/run/
Common configuration analysis

Super brother's salt master file

[root@master ~]# grep -v ^# /etc/salt/master|grep -v ^$
interface:  #Bind to local address
publish_port: 4505  #Management port, command sending
user: root      #Users running the salt process
worker_threads: 5  #The number of threads run by salt. The more threads, the faster the processing speed, and do not exceed the number of CPUs
ret_port: 4506  #Execution result return port
pidfile: /var/run/ #pid file location
log_file: /var/log/salt/master  #Log file address

#Automatically receive the key of minion
auto_accept: False
# minion Recognition ID,Could be IP,Domain name, or through DNS Parsed string
id: slave
# salt User rights to run
user: root
# master Recognition ID,Could be IP,Domain name, or through DNS Parsed string
master : master
# master Communication port
master_port: 4506
# Backup mode, minion It is a local backup, the file backup mode when file management is performed
backup_mode: minion
# implement salt-call Time output mode
output: nested
# minion wait for master Time of certification
acceptance_wait_time: 10
# The number of failed reconnections, 0 means infinite times, non-zero will continue to try to the set value and stop trying
acceptance_wait_time_max: 0
# The delay time of recertification can be avoided because master Of key Change leads to minion Requiring recertification syn Storm
random_reauth_delay: 60
# Log file location
log_file: /var/logs/salt_minion.log
Common configuration of salt minion

Satl minion file of super brother

[root@slave ~]# grep -v ^# /etc/salt/minion|grep -v ^$
master: master
master_port: 4506
user: root
id: slave
acceptance_wait_time: 10
log_file: /var/log/salt/minion

Start salt master and salt minion

systemctl start salt-minion
systemctl start salt-master

#Check salt status
systemctl status salt-minion
systemctl status salt-master

Receive minion key on master

After minion is started, the master will ask the master to sign a certificate for it. After the certificate is signed, the master can trust minion, and the communication between minion and master is encrypted.

Execute in salt master

The salt key command is used to manage the mionion secret key

[root@master ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
slave    #Now slave has appeared
Rejected Keys:

At this time, slave has appeared in unaccepted keys, indicating that minion has contacted the master, and the master has obtained the public key of minion, waiting for the next instruction.

Check the key match between master and minion

Execute on master

[root@master ~]# salt-key -f slave
Unaccepted Keys:
slave:  0a:b9:97:b5:9a:65:b8:49:a1:50:a5:6a:66:ce:33:ea

You can then obtain the secret key of minion on minion

[root@slave ~]# salt-call --local key.finger

Therefore, you can confirm the key match and receive the key on the master

[root@master ~]# salt-key -a slave
The following keys are going to be accepted:
Unaccepted Keys:
Proceed? [n/Y] y
Key for minion slave accepted.

After confirming the receipt of the secret key, check whether the minion secret key is received

[root@master ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:

After receiving the secret key, you can execute the first salt


Only after Master accepts Minion Key can it be managed. The specific authentication commands for salt key are as follows.

[root@linux-node1 ~]# salt-key -L
Accepted Keys:   #Accepted key
Denied Keys:     #Rejected key
Unaccepted Keys: #key not added
Rejected Keys: #Revoked key

#Common parameters
-L  #View KEY status
-A  #Allow all
-D  #Delete all
-a  #Verify the specified key
-d  #Delete the specified key
-r  #Log off the specified key (the status is not authenticated)

#At the master end/etc/salt/master To configure
auto_accept: True   #If you trust Minion, you can configure master to automatically accept requests

Use of salt key

#List all current key s
[root@salt0-master ~]# salt-key 
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:

#Add the key of the specified minion
[root@salt0-master ~]# salt-key  -a -y
#Add key of all minion s
[root@salt0-master ~]# salt-key  -A  -y

#Delete the specified key
[root@salt0-master ~]# salt-key -d -y
#Delete all key s
[root@salt0-master ~]# salt-key -D -y


Daily command parameters

First know what files are installed in the master and minion, then know how to operate

master terminal

rpm -ql salt-master
/etc/salt/master      # salt master master master profile
/usr/bin/salt           #salt master core operation command
/usr/bin/salt-cp       #salt file transfer command
/usr/bin/salt-key    #salt certificate management
/usr/bin/salt-master    #salt master service command
/usr/bin/salt-run          #salt master runner command

slave terminal

[root@slave  ~]$rpm -ql salt-minion
/etc/salt/minion     #minion profile
/usr/bin/salt-call    #Pull command
/usr/bin/salt-minion   #minion service command
/usr/lib/systemd/system/salt-minion.service   #minion startup script

First salt command

[root@master ~]# salt '*'

# salt It's an order
# * Represents the target host, Represent all target hosts here
# is the method below a module executed remotely by salt.

This is a very simple command to detect the survival of minion host, which is also a remote execution command. We send a message to "*" all minions through the master, and tell them to run the built-in command of salt (also a function in the python module), and return true to indicate that the slave machine monitors the survival.


The test module actually has many other functions

[root@master ~]$salt '*' sys.list_functions test
    - test.arg
    - test.arg_repr
    - test.arg_type
    - test.assertion
    - test.attr_call
    - test.collatz
    - test.conf_test
    - test.cross_test
    - test.echo
    - test.exception
    - test.fib
    - test.get_opts
    - test.kwarg
    - test.module_report
    - test.not_loaded
    - test.opts_pkg
    - test.outputter
    - test.provider
    - test.providers
    - test.rand_sleep
    - test.rand_str
    - test.retcode
    - test.sleep
    - test.stack
    - test.try_
    - test.tty
    - test.version
    - test.versions_information
    - test.versions_report
test other functions

test.echo under test

[root@master ~]$salt '*' test.echo 'tiger nb666'
    tiger nb666

test.fib generate Fibonacci sequence

Fibonacci sequence is defined as 0 in item 0 and 1 in Item 1. The sequence starts from item 3, and each item is equal to the sum of the first two items.

[root@master ~]$salt '*' test.fib 50
      - 0
      - 1
      - 1
      - 2
      - 3
      - 5
      - 8
      - 13
      - 21
      - 34
    - 1.09672546387e-05


Previously, salt minion and salt master were installed on the machine for the simplest connection. Then the master accepted the secret key of minion and ran the first command

salt command structure

The commands entered at the command line are all execution modules

When a command is written to a file, it is called a status module

salt --help #To view salt help

[root@master ~]$salt --help
Usage: salt [options] '<target>' <function> [arguments]

salt Command parameter target salt Parameters for remote execution of function of module

Complete five part command

#--The summary parameter displays a summary of the salt command
[root@master ~]$salt --summary '*' 'hostname' slave: slave ------------------------------------------- Summary ------------------------------------------- # of Minions Targeted: 1 # of Minions Returned: 1 # of Minions Did Not Return: 0 -------------------------------------------

List sys modules for all salt

#sys module interacting with the system
[root@master ~]$salt 'slave' sys.list_modules

Remote execution command module

cmd is a super module. All shell commands can be executed

[root@master ~]$salt 'slave' 'ps -ef|grep python'
    root        905      1  0 07:31 ?        00:00:02 /usr/bin/python -Es /usr/sbin/tuned -l -P
    root       3843      1  0 11:05 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion
    root       3846   3843  0 11:05 ?        00:00:01 /usr/bin/python /usr/bin/salt-minion
    root       4031      1  0 11:31 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion
    root       4032   4031  0 11:31 ?        00:00:00 /bin/sh -c ps -ef|grep python
    root       4034   4032  0 11:31 ?        00:00:00 grep python



Remote installation of nginx

#Installing nginx on minion
[root@master ~]$salt 'slave' pkg.install "nginx"
#Uninstall nginx on minion
[root@master ~]$salt 'slave' pkg.remove "nginx"
#Check pkg package version
[root@master ~]$salt 'slave' pkg.version "nginx"

Remote management service module

Management service is an important task for system administrator. It is very easy to manage minion service through salt. Use service module

[root@master ~]$salt 'slave' service.start "nginx"
[root@master ~]$salt 'slave' service.status "nginx"
[root@master ~]$salt 'slave' service.stop "nginx"


Like standard Linux commands, salt commands are used the same way

--out controls the format of the result output of the salt command


[root@master ~]$salt --out=json '*' cmd.run_all 'hostname'
    "slave": {
        "pid": 2268,
        "retcode": 0,
        "stderr": "",
        "stdout": "slave"


[root@master ~]$salt --out=yaml '*' cmd.run_all 'hostname'
  pid: 2289
  retcode: 0
  stderr: ''
  stdout: slave

YAML explanation

In the process of learning saltstack, the first point is the writing skills of States, which is called SLS file for short. This file follows YAML syntax. It's easy for beginners to be confused. Come on, super brother saves you from learning YAML grammar
 json xml yaml data serialization format
 yaml is easy to parse and applied to configuration files

The configuration file of salt is yaml. You cannot use tab
 yaml format configuration file used by saltstack, k8s and ansible

Rule of grammar
    Case sensitive
    Use indents to represent hierarchy   
    tab key is forbidden when indenting, only space is allowed
    The number of indented spaces is not important. The left side of elements of the same level
    #Represents a comment line
 Data structure supported by yaml
    Object: key value pair, also known as mapping hash hashes dictionary dict colon indicates that there must be
    Array: a set of values arranged in order, also known as sequence list short bar - list1
    Pure quantity: a single value that cannot be subdivided

Objects: key value pairs



YAML yes YAML Ain't Markup Language The acronym for, YAML Its grammar is simple,
Structure is displayed by spaces
Project use '-' representative
Pass key value pair ':' Division
YAML syntax follows a fixed indentation style, indicating the hierarchical relationship of data. saltstack requires that each indentation level is composed of 2 spaces, and tabs are not allowed!!!
The dictionary in Python is a simple key value pair, which is called hash table map in go
Dictionary key s are separated by colons

The expression of key in YAML is a string of colon result
my_key: my_value
Convert to python syntax. The above command is

value can also be associated with key through indentation, four spaces!!
The same goes for converting to python syntax

Dictionaries can be nested in YAML syntax
Convert to python syntax


Short bar
 YAML syntax represents a list with a bar and a space
 Multiple items use the same indentation level as part of the same list
- list_value_one
- list_value_two
- list_value_three
List can be used as the value of a key value pair, for example, to install multiple software at one time
- l1
- l2
- l3
The understanding of converting to python code is


Target location string

In the previous salt commands, we used salt '*' to control all minions, and we only have one "slave", but the servers in the production environment are likely to be hundreds of minions, so we need to flexibly locate the required servers and execute remote commands.


1.Global matching
[root@master ~]$salt '*'
Linux wildcard
*               Represents any character or empty string
?               Represents a character, cannot be empty
[a-z] [0-9]     Represents any lowercase letter

[root@master ~]$salt 'slav?'

[root@master ~]$salt '[a-z]lave'
salt Sea support python Of re regular expression

The state module defines the host state

The previously executed remote commands are all procedural, similar to the execution of a shell or python script, which triggers the same function once.

So a large number of remote commands running on minion must use the "state management" provided by salt. State is a description and definition of minion. Operation and maintenance do not need to care about how the deployment is completed, but only need to describe what state minion needs to achieve.

Next, deploy nginx through the state module

master terminal

vim /etc/salt/master 
#Open the comment, write the following, strictly follow the syntax, space file_roots: base: - /srv/salt/base dev: - /srv/salt/dev test: - /srv/salt/test prod: - / srv/salt/prod

#This step needs to be executed in both the master and minion, folders and nginx.sls files
[root@master  ~]$mkdir -p /srv/salt/{base,dev,test,prod}

[root@slave   ~]$mkdir -p /srv/salt/{base,dev,test,prod}

#The nginx.sls status file must be available on both the master and minion
[root@master /srv/salt/base]$cat nginx.sls nginx-install: pkg.installed: - name: nginx nginx-service: service.running: - name: nginx - enable: True
Explain the nginx.sls description file
sls configuration files are described in YAML The first command uses the pkg.install command, which is relative to running yum install. At this time, the state module will determine whether nginx is installed. If it is not installed, install it, and do nothing if it is installed. The state description will follow this principle. Only when the real state is detected and the required state is different, this function will be executed. This property is called idempotence.

At this time, we use the state module to deploy nginx software. Through the nginx.sls descriptive configuration file we wrote, the command line calls the SLS function of the state module.

#Start the command. At this time, the slave has been installed and nginx is alive for state management
[root@master  /srv/salt/base]$salt 'slave' state.sls nginx

You can go to the slave test to shut down nginx, delete nginx, and execute the command again. You can also install and start nginx

Check nginx status of slave through master

[root@master /srv/salt/base]$salt 'slave' 'ps -ef|grep nginx'
    root       8543      1  0 Sep14 ?        00:00:00 nginx: master process /usr/sbin/nginx
    nginx      8544   8543  0 Sep14 ?        00:00:00 nginx: worker process
    root       8592   8591  0 00:03 ?        00:00:00 /bin/sh -c ps -ef|grep nginx
    root       8594   8592  0 00:03 ?        00:00:00 /bin/sh -c ps -ef|grep nginx

Grains of static information collected by Salt

If you are employed, your boss asks you to collect hardware information about 300 servers in the company. Are you a login? Or choose to use salt to collect? Or write scripts with python's salt API

Grains is one of the most important components of saltstack. They are often used in configuration and deployment. Grains records the static information of minion, such as common properties, CPU, memory, disk, network information, etc.
The Grains information of Minions is collected and reported to Master when Minion is started
Grains is a database stored in the form of key value, which can be regarded as the metadata of the Host
Grains holds the collected client details
If the slave machine data changes, grains will be out of date You need to customize Grains in the production environment. You can use the Minion profile Grains related module definition Python script definition

salt 'slave' sys.doc grains view the command usage of grains


Grains describes the static attribute data inherent in minion and lists all grains data of the host

[root@master ~]$salt 'slave' grains.items

Information too long, omitted

salt 'slave' ා list all grains methods

Retrieve some data

[root@master ~]$salt 'slave' grains.item os id host

Using Grains static information to locate the host

Before super brother, the company had 100 + redhat operating system and 80 + centos. Before knowing salt, it was very ignorant..

In addition to different systems, there are different system versions, RedHat 6. X CentOS 6. X

There are two ways of writing:
salt '*' grains.item key1 key2 key3
salt '*' -G

#Positioning the Cenots machine
[root@master ~]$salt -G 'os:CentOS' slave: True
#The positioning operating system is a 7-series machine

[root@master ~]$salt -G 'osrelease:7*'

#Find out the ip address

salt '*' grains.item fqdn_ip4

Therefore, use grains.items to list all data matching hosts and locate data according to a single information. Grains can also be customized to meet different needs.

Custom settings Grains data

#Setting data
[root@master ~]$salt 'slave' grains.setval cpu_num 8 slave: ---------- cpu_num: 8
#Query data
[root@master ~]$salt 'slave' grains.item cpu_num slave: ---------- cpu_num: 8

Set the Grains static data on the master side. The principle will add this data to the / etc/salt/grains of the configuration file of the minion server

[root@slave ~]$cat /etc/salt/grains
cpu_num: 8

For complex data structures, flexible JSON syntax can be added

[root@master ~]$salt 'slave' grains.setval cpu_info '["Intel","Xeon","10"]'
        - Intel
        - Xeon
        - 10
[root@master ~]$salt 'slave' grains.item cpu_info
        - Intel
        - Xeon
        - 10

At this time, you can check the grains file on the minion server

[root@slave ~]$cat /etc/salt/grains
- Intel
- Xeon
- '10'
cpu_num: 8

Therefore, after Grains data is written to the configuration file, restart the salt minion service, and the data will not be lost

If you want to delete it, you can delete it by using the command, or delete the configuration file of demiion (or delete the file)

1.Method 1, clear the value
[root@master ~]$salt 'slave' grains.delval cpu_info
[root@master ~]$salt 'slave' grains.delval cpu_num

2.Method 2 delete minion Of grains Profile, restart service

[root@slave ~]$rm -rf /etc/salt/grains
[root@slave ~]$!sys
systemctl restart salt-minion

Check result deleted successfully

Pillar component of data management center

Pillar is also one of the most important components of saltstack, which is called data management center. It is often used in large-scale configuration management with states.

The Pillar is dynamic, stored in the master side, provided and provided to the minion side

Pillar's main role in SaltStack is to store and define some data needed in configuration management, such as software version number, user account password, etc., so as to ensure that sensitive data is not seen by other minion s

The storage format is similar to Grains, which is YAML format

In the master configuration file, there is a section of Pillar settings options that specifically define the parameters of Pillar


cat /etc/salt/master  
#This configuration represents the working root directory of pillar. Under / srv/pillar, you can create a new sls file #pillar_roots: # base: # - /srv/pillar

By default, the working directory of pillar is in / srv/pillar directory. Execute the following code

mkdir -p /srv/pillar
#Specify environment, mark, reference packages.sls and services.sls
vim /srv/pillar/top.sls
      - packages
      - services
vim /srv/pillar/packages.sls nginx: packages-name: nginx version: 1.12.2 vim /srv/pillar/packages.sls nginx: port: 80 user: root

Check the pilar value we set

[root@master /srv/pillar]$salt '*' pillar.item nginx

Pillar vs. Grains

pillar vs. Grains

Type data collection method application scenario definition location
 Grains static minion collection data query target selection configuration management minion
 Pillar dynamic master for custom target selection configuration management sensitive data master

Python API calls SaltStack

SaltStack itself provides salt(usr/bin/salt) for interactive management, but it's certainly not a long way to go to the server and type commands. At this time, python embodies the function of nb.

Python API is the API provided for Python. It needs to be run on SaltStack master

Example code

[root@master  ~]$python
Python 2.7.5 (default, Jul 13 2018, 13:06:57)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import salt.client
>>> local = salt.client.LocalClient()  #<salt.client.LocalClient object at 0x7f886dbdd850>
>>> local.cmd('*','',['hostname'])#Send commands to all minion s
{'slave': 'slave'}

So the python API provides an entry to send commands to saltstack.

Get the configuration file of saltstack through API

Get master profile

>>> import salt.config #Import salt configuration module
>>> m_opts=salt.config.client_config('/etc/salt/master') #Read the salt configuration file to get a dictionary data

Get minion profile

Python 2.7.5 (default, Jul 13 2018, 13:06:57)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import salt.client
>>> salt.config.minion_config('/etc/salt/minion') #Read minion configuration file, get dictionary data, and view information through dictionary method

Introduction to Python API

/usr/bin/salt The default interface used is LocalClient,This interface can only be used in salt master Upper use
[root@master  ~]$python
Python 2.7.5 (default, Jul 13 2018, 13:06:57)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import salt.client
>>> local = salt.client.LocalClient()  #<salt.client.LocalClient object at 0x7f886dbdd850>
>>> local.cmd('*','',['hostname'])#Send commands to all minion s
{'slave': 'slave'}

Return results one by one, local. CMD writer()

>>> ret=local.cmd_iter('*','')
>>> ret
<generator object cmd_iter at 0x7f886d455c80>
>>> for i in ret:
...     print i
{'slave': {'retcode': 0, 'ret': True}}

Fibonacci series

>>> local.cmd('*','test.fib',[10])
{'slave': [[0, 1, 1, 2, 3, 5, 8], 2.1457672119140625e-06]}

Check minion server information

>>> local.cmd('*','',['hostname'])
{'slave': 'slave'}
>>> local.cmd('*','',['ifconfig'])
>>> local.cmd('*','',['crontab -l'])
>>> local.cmd('*','',['df -h'])

Start stop minion services, such as nginx

>>> local.cmd('*','service.stop',['nginx'])
{'slave': True}
>>> local.cmd('*','service.status',['nginx'])
{'slave': False}
>>> local.cmd('*','service.start',['nginx'])
{'slave': True}


Posted by digioz on Sat, 11 Jan 2020 00:33:32 -0800