Daemon

Main Process Create Daemon (Subprocess)

Then the main process is the daemon

• The daemon terminates after the main process code executes
• The daemon cannot open any more child processes or an exception is thrown: AssertionError: daemonic processes are not allowed to have children

Note: Processes are independent. The main process code runs and the daemon terminates.

from multiprocessing import Process
import time

def task():
    print('sub start')
    print('sub over')


if __name__ == '__main__':
    p =Process(target=task)
    p.daemon = True  # Setting this process as a child process must precede p.start()
    p.start()
    print('parent start')
    time.sleep(1)   # The reason for this delay is to let the child process run. Otherwise, the child process (the daemon) will not run completely, and the main program (the daemon) will run and end the code.
    print('parent over')
    
# If there is no delay here in the main program, the sub-process will not be printed because the main program code has been run when the operating system starts the sub-process.   
parent start
sub start  
sub over
parent over

Process security issues

Because the memory of the child processes is isolated from each other, data is not shared between processes; if multiple sub-processes access the same file at the same time, or print the same terminal, there will be competition, which results in confusion, which is a process security problem.

Processes 1 and 2 are here, so the range is set so wide that problem printing can be shown

from multiprocessing import Process

def task1():
    for i in range(10000):
        print('sub1 run')
        print('sub1  over')


def task2():
    for i in range(10000):
        print('sub2 run')
        print('sub2  over')


if __name__ == '__main__':
    p1 = Process(target=task1)
    p2 = Process(target=task2)

    p1.start()
    p2.start()

    
sub1  over
sub2  over
sub1 run
sub2 run
sub1  over
sub2  over

Mutex (process synchronization)

What is mutex?

Mutually exclusive locks

Principle:

The idea is to lock code that operates on a public resource so that only one process can execute it at a time

Data is not shared between processes, but sharing the same file system, accessing the same file at the same time, or printing the same terminal will result in competition, which results in confusion. How to control it is to lock the process

In order to solve the above process security problems caused by sub-process competition, we use mutex to solve this problem.

from multiprocessing import Process,Lock

def task1(mutex):
    mutex.acquire()
    for i in range(10000):
        print('sub1 run')
        print('sub1  over')
        
    mutex.release()

def task2(mutex):
    mutex.acquire()
    for i in range(10000):
        print('sub2 run')
        print('sub2  over')
    mutex.release()


if __name__ == '__main__':
    mutex = Lock()
    p1 = Process(target=task1,args=(mutex,))
    p2 = Process(target=task2,args=(mutex,))

    p1.start()
    p2.start()

Principle of locks

def task1():
    global lock
    if lock == False:
        lock = True
        open("aaa.txt","wt")
        lock = False


def task2():
    global lock
    if lock == False:
        lock = True
        open("aaa.txt","wt")
        lock = False

As you can see, locks don't lock data out of use, they just keep code from executing.

manager Manager

Multiple sub-processes read and write a shared file at the same time. Although locked, it can sometimes cause file confusion because the data between sub-processes is not interchangeable, that is, the data is not synchronized. Then we need to create a synchronization channel.

from multiprocessing import Process,Manager,Lock
import time

def task(data,lock):
    lock.acquire()
    num = data[0]
    time.sleep(0.2)
    data[0] = num -1
    lock.release()


if __name__ == '__main__':
    d = [100]
    m = Manager()  # Create a manager
    sync_list = m.list(d)  # Let Manager create a list of process synchronizations (or dictionaries, etc.)

    lock = Lock() # Create a lock

    ps = []
    for i in range(10):
        p = Process(target=task,args=(sync_list,lock))
        p.start()
        ps.append(p)

    for p in ps:p.join()

    print(d)
    print(sync_list)
    
sub over
sub over
sub over
sub over
sub over
sub over
sub over
sub over
sub over
sub over
[100]
[90]

Summary:

Locking ensures that when multiple processes modify the same piece of data, only one task can be modified at the same time, that is, concurrently to serially, which ensures data security at the expense of speed.

Although file sharing data can be used for interprocess communication, the problem is:

• Inefficient: Shared data is file-based and files are on the hard disk
• Need to lock yourself

This method is suitable for situations where interaction is infrequent and data is large.

It is not appropriate to have a small amount of data due to frequent interactions, so we use another solution: IPC (Interprocess Communication) - Queue + Pipeline

queue

Processes are isolated from each other. To achieve interprocess communication (IPC), the multiprocess module supports two forms: queues and pipelines, both of which use messaging.

Queue is FIFO

from multiprocessing import Queue

q = Queue(2)  # Create a queue and store only two elements at the same time, defaulting to infinity if not written
q.put(1)  # Store data in pipeline
q.put(2)
# q.put(3,block=False,timeout=3)  # block=True means blocking, defaulting to True; timeout=3 means waiting delay 3s, defaulting to None;
                               # When there is no place in the container, it blocks and waits for 3 seconds for someone to remove an element from the pipe.
                               # Then element 3 will be saved in, otherwise error queue.Full will be thrown

print(q.get())  # get is to take data out and be FIFO
print(q.get())
# print(q.get(block=True,timeout=3)) # The default is blocking until someone saves an element, just like saving it.