1.1 Processes and Procedures

Written code, when not running, is called code

Running code, called processes

In addition to containing code, a process must have a running environment for the code.

1.2 fork () [windows does not support fork]

The os.fork() method executes once and returns twice. The operating system automatically assigns the current process (parent process) to one (child process), and then executes within the parent process and the child process, respectively.

Subprocess returns 0

The parent process returns the id of the child process

import os
pid = os.fork()
print("do")
if pid == 0:
    print("child process pid %d" % pid)
else:
    print("parent process pid %d" % pid)

getpid()   getppid()

import os
pid = os.fork()
print("do")
if pid == 0:
# print("child process id %d" % pid)
    print("current child process pid %d parent pid %d" % (os.getpid(), os.getppid()))
else:
# print("parent process id %d" % pid)
    print("current parent process pid %d child pid %d" % (os.getpid(), pid))

A child process can create a child process, and the execution of the parent and child processes is not sequential.

Each process in a multi-process replicates all the data of the parent process without affecting each other.

Each process in a multi-process replicates all the data of the parent process without affecting each other.
import os
pid = os.fork()
num = 1
print("do", num)
if pid == 0:
    num+=1
    print(num)
else:
    num+=1
    print(num)

Output results:

do 1
2
do 1
2

1.3 multiprocessing Module

from multiprocessing import Process
import os
def fun():
    print("child pid %d" % os.getpid())


print("parent pid %d" % os.getpid())
p = Process(target=fun)
print("has not exec")
p.start()
p.join()
print("exec finish")

The grammatical structure of Process is as follows:

Process([group [, target [, name [, args [, kwargs]]]]])

target: Represents the object invoked by the process instance;

args: a tuple of location parameters representing the calling object;

kwargs: a keyword parameter dictionary representing the calling object;

name: an alias for the current process instance;

Common methods of the Process class:

is_alive(): Determines whether the process instance is still executing;

join([timeout]): Whether to wait for the end of process instance execution, or how many seconds to wait;

start(): Start a process instance (create a subprocess);

run(): If no target parameter is given, when the start() method is called on this object, the run() method in the object is executed.

terminate(): terminate immediately, regardless of whether the task is completed or not;

Common attributes of the Process class:

name: The current process instance alias, default to Process-N, N is an integer incremental from 1;

Pid: The PID value of the current process instance (int value after the start call)

1.4 Process Pool

When initializing a process pool, you need to specify the maximum number of processes (the maximum number of processes must not be greater than the number of cpu cores). When requesting a Pool, you will retrieve the process according to the current pool, if available.

Execute with a process, wait until there are available processes if there are no available processes

from multiprocessing import Pool
import os
import time
def fun(name, age, **kwargs):
    print("pid %d" % os.getpid())
    print(name, age, kwargs)
    time.sleep(1)
pool = Pool(5)
for i in range(20):
    pool.apply_async(fun, ("zzy", 20), {"isalive": False})
pool.close()
pool.join()
print("finish")

multiprocessing.Pool common function analysis:

apply_async(func[, args[, kwds]]): Call func in a non-blocking manner (and execute, blocking mode must wait for the previous process to exit before executing the next entry

args is the parameter list passed to func and kwds is the keyword parameter list passed to func.

apply(func[, args[, kwds]]): Call func in a blocking manner

close(): Close Pool so that it no longer accepts new tasks, but it affects the running process;

terminate(): terminate immediately, regardless of whether the task is completed or not;

join(): the main process is blocked, waiting for the exit of the child process, which must be used after close or terminate;

from multiprocessing import Pool
import os
import time
def fun(name, age, **kwargs):
    print("pid %d" % os.getpid())
    time.sleep(1)
pool = Pool(5)
for i in range(20):
    pool.apply_async(fun, ("zzy", 20), {"isalive": False})
pool.close()
time.sleep(1)
pool.terminate()
# pool.join()
print("finish")

1.5 Queue Interprocess Communication

Common methods:

put(): Putting data into a queue will block, equivalent to put(item,True)

get(): fetching data from a queue will block, equivalent to get(item,True)

put_nowait(): Equivalent to put(item, False), no blocking

get_nowait(): Equivalent to get(item, False), no blocking

size(): Gets the queue length

empty(): Determines whether the queue is empty

full(): Determine whether the queue is full

Sharing data using Queue:

from multiprocessing import Process
from multiprocessing import Pool
from multiprocessing import Queue
from multiprocessing import Value
import time
def read(q):
    while True:
        print("pqueue size %d" % q.qsize())
        r = q.get()
        print("get %d" % r)
        print("pqueue size %d" % q.qsize())
def write(q):
    for r in range(5):
        q.put(r)
        print("put %d " % r)
        print("queue size %d" % q.qsize())
        time.sleep(2)
q = Queue(10)
q.put(-1)
q.put(-2)
pr = Process(target=read, args=(q,))
pw = Process(target=write, args=(q,))
pr.start()
pw.start()
pr.join()
pw.join()
print("finish")

Sharing data under the process pool:

from multiprocessing import Pool
from multiprocessing import Manager
import time
def read(q):
    while True:
        print("pqueue size %d" % q.qsize())
        r = q.get()
        print("get %d" % r)
        print("pqueue size %d" % q.qsize())
        time.sleep(0.1)
def write(q):
    for r in range(5):
        q.put(r)
        print("put %d " % r)
        print("queue size %d" % q.qsize())
        time.sleep(2)
q = Manager().Queue(5)
q.put(-2)
q.put(-1)
pool = Pool(5)
pool.apply_async(write, (q,))
pool.apply_async(read, (q,))
pool.close()
pool.join()
print("finish")