Several methods commonly used in python multiprocess programming

Keywords: Python Attribute

Multi-threading in Python is not really multi-threading. If you want to make full use of multi-core CPU resources, in most cases you need to use multi-processes in python.Python provides a very useful multiprocess package, Multiprocessing, that just defines a function and python does everything else.With this package, the conversion from single process to concurrent execution can be easily completed.Multprocessing supports subprocesses, communicating and sharing data, performing different forms of synchronization, and provides Process, Queue, Pipe, LocK, and other components

1. Process

Syntax: Process([group[,target[,name[,args[,kwargs]]]])

Parameter meanings: target denotes the calling object; args denotes the location parameter meta-ancestor of the calling object; kwargs denotes the dictionary of the calling object.The name is an alias and groups will not actually be called.

Method: is_alive():

   join(timeout):

   run():

   start():

   terminate():

Attributes: authkey, daemon (to be set via start(), exitcode (process is None at run time, if -N, it is ended by signal N), name, pid.Where daemon is automatically terminated after the parent process terminates and cannot generate a new process by itself, it must be set before start().

1. Create a function as a single process

from multiprocessing import Process
def func(name):
    print("%s Once a good person"%name)

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    p.start()   #start() tells the system to start the process

2. Create functions and use them as multiple processes

from multiprocessing import Process
import random,time

def hobby_motion(name):
    print('%s Like Sports'% name)
    time.sleep(random.randint(1,3))

def hobby_game(name):
    print('%s Like games'% name)
    time.sleep(random.randint(1,3))

if __name__ == "__main__":
    p1 = Process(target=hobby_motion,args=('Fu Tingting',))
    p2 = Process(target=hobby_game,args=('Kobe',))
    p1.start()
    p2.start()

Execution results:

Fu Tingting likes sports
 Kobe likes games

3. Define a process as a class (another way to start a process, which is not very common)

from multiprocessing import Process
class MyProcess(Process):
    def __init__(self,name):
        super().__init__()
        self.name = name

    def run(self):  #When start(), run is called automatically, and can only be defined here as run.
        print("%s Once a good person"%self.name)

if __name__ == "__main__":
    p = MyProcess('kebi')
    p.start()  #Process is treated as a parent class and a function is customized.

4. Comparison of daemon programs

Without daemon attribute

import time
def func(name):
    print("work start:%s"% time.ctime())
    time.sleep(2)
    print("work end:%s"% time.ctime())

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    p.start()
    print("this is over")

#results of enforcement
this is over
work start:Thu Nov 30 16:12:00 2017
work end:Thu Nov 30 16:12:02 2017

Add the daemon attribute

from multiprocessing import Process
import time
def func(name):
    print("work start:%s"% time.ctime())
    time.sleep(2)
    print("work end:%s"% time.ctime())

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    p.daemon = True   #Parent process terminates automatically after termination, no new process can be generated, must be set before start()
    p.start()
    print("this is over")

#results of enforcement
this is over

The method you want to execute once you have set the daemon property:

import time
def func(name):
    print("work start:%s"% time.ctime())
    time.sleep(2)
    print("work end:%s"% time.ctime())

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    p.daemon = True
    p.start()
    p.join()  #Execute the previous code before executing the following
    print("this is over")

#results of enforcement
work start:Thu Nov 30 16:18:39 2017
work end:Thu Nov 30 16:18:41 2017
this is over

5.join(): The code above will not execute until it is executed.

Let's start with an example:

from multiprocessing import Process
import time,os,random
def func(name,hour):
    print("A lifelong friend:%s,%s"% (name,os.getpid()))
    time.sleep(hour)
    print("Good bother:%s"%name)

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',2))
    p1 = Process(target=func,args=('maoxian',1))
    p2 = Process(target=func,args=('xiaoniao',3))
    p.start()
    p1.start()
    p2.start()
    print("this is over")

Execution results:

this is over   #Last executed, first printed, indicating that start() only starts the process, not necessarily completes it
A lifelong friend:kebi,12048
A lifelong friend:maoxian,8252
A lifelong friend:xiaoniao,6068
Good bother:maoxian   #Print first, execute second
Good bother:kebi     
Good bother:xiaoniao

Add join ()

from multiprocessing import Process
import time,os,random
def func(name,hour):
    print("A lifelong friend:%s,%s"% (name,os.getpid()))
    time.sleep(hour)
    print("Good bother:%s"%name)
start = time.time()
if __name__ == "__main__":
    p = Process(target=func,args=('kebi',2))
    p1 = Process(target=func,args=('maoxian',1))
    p2 = Process(target=func,args=('xiaoniao',3))
    p.start()
    p.join()   #After executing the above code, execute the following
    p1.start()
    p1.join()
    p2.start()
    p2.join()
    print("this is over")
    print(time.time() - start)

#results of enforcement
A lifelong friend:kebi,14804
Good bother:kebi
A lifelong friend:maoxian,11120
Good bother:maoxian
A lifelong friend:xiaoniao,10252  #Each process is finished before the next one is executed
Good bother:xiaoniao
this is over
6.497815370559692   #2+1+3+Main Program Execution Time

Change location

from multiprocessing import Process
import time,os,random
def func(name,hour):
    print("A lifelong friend:%s,%s"% (name,os.getpid()))
    time.sleep(hour)
    print("Good bother:%s"%name)
start = time.time()
if __name__ == "__main__":
    p = Process(target=func,args=('kebi',2))
    p1 = Process(target=func,args=('maoxian',1))
    p2 = Process(target=func,args=('xiaoniao',3))
    p.start()
    p1.start()
    p2.start()
    p.join()   #2 seconds required
    p1.join()  #Execution is now complete
    p2.join()   #Executed for 2 seconds, 1 second
    print("this is over")
    print(time.time() - start)

#results of enforcement

A lifelong friend:kebi,13520
A lifelong friend:maoxian,11612
A lifelong friend:xiaoniao,17064  #Almost simultaneously open execution
Good bother:maoxian
Good bother:kebi
Good bother:xiaoniao
this is over
3.273620367050171  #Give priority to the longest time

6. Other attributes and methods

from multiprocessing import Process
import time
def func(name):
    print("work start:%s"% time.ctime())
    time.sleep(2)
    print("work end:%s"% time.ctime())

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    p.start()
    p.terminate()  #Kill the process and must be placed behind start(), similar to what daemon does

#results of enforcement
this is over
from multiprocessing import Process
import time
def func(name):
    print("work start:%s"% time.ctime())
    time.sleep(2)
    print("work end:%s"% time.ctime())

if __name__ == "__main__":
    p = Process(target=func,args=('kebi',))
    # p.daemon = True
    print(p.is_alive())
    p.start()
    print(p.name)   #Get the name of the process
    print(p.pid)    #Get pid of process
    print(p.is_alive())  #Determine whether a process exists
    print("this is over")

Posted by seby on Sat, 04 Jan 2020 06:01:07 -0800