shell script multitasking concurrent execution

Transferred from: https://blog.51cto.com/yttitan/2409618

Normally, the commands in the Shell script are executed serially. When a command is executed, the next command will be executed. For example, the following code:

#!/bin/bash
for i in {1..10};do
echo $i 
done
echo "END"

Execution results:

1
2
3
4
5
6
7
8
9
10
END

As you can see, the "echo $i" command in the loop body is executed serially. However, if the executed command takes a long time, it will lead to a very long execution time of the whole program, and may even cause the program to get stuck and lose response for a long time.

For example, we need to complete such a task: write a script to scan the 192.168.80.0/24 network. The hosts that are currently online are considered online if they can ping.

To complete this task, writing a script is not complicated. Here is the written code:

#!/bin/bash
for i in {1..254};do
        ip="192.168.80.$i"
        ping -c 2 $ip &> /dev/null && echo $ip is up 
done

Here is a brief description of the ping command used in the script. The ping command in Linux will send out packets continuously after execution. Therefore, the "- c" option is used in the ping command in the script to specify that packets are sent only twice. If a response can be received, the target host is considered to be online.

There is no logical problem with this script, but after execution, it takes a long time to ping 254 IP addresses in the network in turn. Moreover, the script cannot be forcibly terminated by Ctrl+C. you can only use Ctrl+Z to transfer to the background, and then use the kill command to forcibly end the process.

[root@localhost ~]# bash ping.sh
192.168.80.1 is up
192.168.80.2 is up
^C
^Z
[1]+  Stopped               bash ping.sh
[root@localhost ~]# jobs -l                                #View background work tasks
[1]+ 101100 stop it                  bash ping.sh
[root@localhost ~]# kill -9 101100                        #Force end process
[root@localhost ~]# 
[1]+  Killed               bash ping.sh

In fact, there is no dependency between the ping commands circularly executed in this script, that is, you don't have to wait until the end of "ping 192.168.80.1" to execute "ping 192.168.80.2". All these ping commands can be executed concurrently.
If Python is used, the concurrent execution of commands can be realized with the help of multithreading technology. However, Shell does not support multithreading, so it can only adopt the multi process method. The specific implementation method is very simple, that is, add "&" after the command to be executed concurrently and transfer it to the background for execution, so that after executing a command, you can immediately transfer to the next command without waiting for its execution to end.
Let's take the previous code as an ex amp le and add "&" after the echo command in the loop body:

#!/bin/bash
for i in {1..10};do
echo $i &
done
echo "END"

Execution results:

[root@localhost ~]# bash test.sh
END
[root@localhost ~]# 1
2
3
6
7
4
8
9
10
5

It can be seen that the execution order of commands cannot be guaranteed during concurrent execution, and the echo "END" command, which should be executed after the execution of the whole cycle, was executed at the beginning of the program. Therefore, during concurrent execution, we usually need to ensure that all commands in the loop body are executed before executing the next command. At this time, we can use the wait command. Using the wait command in the Shell is equivalent to multithreading synchronization in other high-level languages.

The following code is improved by adding the wait command:

#!/bin/bash
for i in {1..10};do
echo $i &
done
wait
echo "END"

In this way, the execution result is normal:

[root@localhost ~]# bash test3.sh
6
7
2
3
4
8
9
10
5
1
END

After understanding the principle of concurrent program execution, we also improved the ping script:

#!/bin/bash
for i in {1..254};do
        ip="192.168.80.$i"
        ping -c 2 $ip &> /dev/null && echo $ip is up &
done
wait

At this point, the execution speed of the script will be greatly improved:

[root@localhost ~]# bash ping.sh
192.168.80.10 is up
192.168.80.20 is up
192.168.80.2 is up
192.168.80.1 is up
192.168.80.135 is up

Therefore, when there is no dependency between the commands to be executed circularly, the method of concurrent execution can be adopted, which can greatly improve the efficiency of code execution. Of course, concurrent execution also has defects, that is, when the number of commands to be executed in parallel is very large, especially when the executed commands occupy a lot of system resources, it may exhaust all the resources of the whole system and affect the operation of other programs. Therefore, other technologies can also be used to limit the number of concurrent execution processes. Due to their complexity, this paper will not introduce them.