python multithreaded programming notes

Keywords: Python

python multithreaded programming notes

Time: August 31, 2021

1, Process

(1) Introduction to multitasking

Multitasking: execute multiple tasks at the same time

Advantages of multitasking: the biggest advantage is to make full use of cpu resources and improve the execution efficiency of programs

Two ways

1. Concurrent

2. Parallel

(2) Introduction to process

Process is the smallest unit of resource allocation. It is the basic unit of resource allocation and scheduling of the operating system (eg: the executing program is a process)

(3) Multi process and multi task

1. Process creation steps

Import process package
- ```
import multiprocessing
```

Creating process objects through process classes

Process object = multiprocressing.Process(target=Task name)

Start the process and execute the task
- ```
Process object.start()
```

#1. Import package
import multiprocessing
import time
def sing():
    for i in range(3):
        print("Singing")
        time.sleep(0.3)
def dance():
    for j in range(3):
        print("Dance")
        time.sleep(0.3)


if __name__ == '__main__':
    #2. Create process
    process_sing = multiprocessing.Process(target=sing)
    process_dance = multiprocessing.Process(target=dance)
    #3. Start process
    process_dance.start()
    process_sing.start()

2. The process executes a task with parameters

Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name

#Import package
import multiprocessing
import time
def sing(num,name):
    print("%s sing"%name)
    for i in range(num):
        print("sing%d......"%i)
        time.sleep(0.3)
def dance(num):
    for j in range(num):
        print("dance%d......"%j)
        time.sleep(0.3)


if __name__ == '__main__':
    #Create process
    # target: task name
    #args: pass parameters to the function as tuples
    #kwargs: pass parameters to functions in dictionary mode
    #Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name
    process_sing = multiprocessing.Process(target=sing,args=(3,"zqq"))
    process_dance = multiprocessing.Process(target=dance,kwargs={"num":4})
    #Start process
    process_dance.start()
    process_sing.start()

(4) . obtain the process number

1. Two ways to get the process number

Get current process number

os.getpid()
Gets the current parent process number

os.getppid()

2. Method of use

Guide Package

import os

#Import package
import multiprocessing
import time
import os
#sing
def sing(num,name):
    print("Singing process pid",os.getpid())  #Current process number
    print("Main process of singing pid",os.getppid())  #Parent process number
    print("%s sing"%name)
    for i in range(num):
        print("sing%d......"%i)
        time.sleep(0.3)
#dance
def dance(num):
    print("The process of dancing pid",os.getpid())
    print("Main process of dancing pid",os.getppid())
    for j in range(num):
        print("dance%d......"%j)
        time.sleep(0.3)


if __name__ == '__main__':
    print("Of the main process pid:%d\n"%os.getpid(),"------"*10)
    #Create process
    # target: task name
    #args: pass parameters to the function as tuples
    #kwargs: pass parameters to functions in dictionary mode
    #Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name
    process_sing = multiprocessing.Process(target=sing,args=(3,"zqq"))
    process_dance = multiprocessing.Process(target=dance,kwargs={"num":4})
    #Start process
    process_dance.start()
    process_sing.start()

(5) . process considerations

1. The main process will wait for the execution of all child processes to end

eg:

import multiprocessing
import time
def work():
    for i in range(10):
        print("At work")
        time.sleep(0.2)
if __name__ == '__main__':
    work_process = multiprocessing.Process(target=work())
   # work_process.daemon = True
    work_process.start()

    time.sleep(1)
    print("End of main process")

2. Set daemon (main process)

Process object. daemon = True

import multiprocessing
import time
def work():
    for i in range(10):
        print("At work")
        time.sleep(0.2)
if __name__ == '__main__':
    work_process = multiprocessing.Process(target=work())
    #Start the main thread guard, the main thread ends, and other threads also end 
    work_process.daemon = True
    work_process.start()
    time.sleep(1)
    print("End of main process")

(6) , multitasking copy manager

1. Demand analysis

Whether the target folder exists. If it exists, it will not be created. If it does not exist, it will be created
Traverse all files in the source file and copy to the destination folder
Multi task is realized by process to complete high concurrent copy

import os
import multiprocessing
def copy_file(file_name,source_dir,dest_dir):
    # 1. Splice source folder path and destination folder path
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    # 2. Open source and destination files
    with open(source_path,"rb") as source_file:   #Read file contents in rb binary mode
        with open(dest_path,"wb") as dest_file:   #wb write file contents in binary mode
    # 3. Loop reading source file to destination path
            while True:
                data =  source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break

if __name__ == '__main__':
    # 1. Define source files and destination folders
    source_dir = r"G:\markdown\python"
    dest_dir = r"D:\python"
    # 2. Create destination folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The target file already exists")
    # 3. Read the file list of the source file
    file_list = os.listdir(source_dir)

    # 4. Traversing the file list to copy
    for file_name in file_list:
        # copy_file(file_list, source_dir, dest_dir)
        # 5. Multi task copy using multi process
        process = multiprocessing.Process(target=copy_file,args=(file_name,source_dir,dest_dir))
        process.start()

2, Thread

(1) Introduction to threads

(2) . multithreading to complete multitasking

1. Steps to create a thread

Import thread module
```
import threading
```

Create thread object

Thread object = threading.Thread(target = Task name)

Start the thread to execute the task
```
Thread object.start()
```

import time
import threading
def sing():
    for i in range(3):
        print("sing:",i)
        time.sleep(1)  #Rest for 1s
def dance():
    for j in range(3):
        print("dance:",j)
        time.sleep(1)

if __name__ == '__main__':
    #sing()
    #dance()
    #Create thread
    sing_thread = threading.Thread(target = sing)
    dance_thread = threading.Thread(target = dance)
    #Open thread
    sing_thread.start()
    dance_thread.start()

(3) . the thread executes tasks with parameters

import time
import threading
def sing(num):
    for i in range(num):
        print("sing:",i)
        time.sleep(1)  #Rest for 1s
def dance(num):
    for j in range(3):
        print("dance:",j)
        time.sleep(1)

if __name__ == '__main__':
    #Create thread
    #args: pay attention to the order of parameters when passing parameters to the function. Parameters are tuples
    #kwargs: pass in the form of dictionary. Note: the parameter name must be consistent with
    sing_thread = threading.Thread(target = sing,args=(3,))
    dance_thread = threading.Thread(target = dance,kwargs={"num":3})
    #Open thread
    sing_thread.start()
    dance_thread.start()

(4) End order of, main thread and child thread

The main thread will wait for the execution of all child threads to end before the main thread ends

There are two ways to set up daemons:

When creating a thread, add the daemon parameter and set it to True
```
threading.Thread(target=Task name,daemon=True)
```
Using damon
```
Thread object.demon = True
```

import threading
import time
def work():
    for i in range(10):
        print("At work",i)
        time.sleep(0.5)
if __name__ == '__main__':
    work_thread = threading.Thread(target=work,daemon=True)
    #Method 1: set daemon thread
    # work_thread.daemon = True
    work_thread.start()
    time.sleep(4)
    print("Main thread end")

(5) Execution order between threads

The order of execution between threads is out of order

Get the current thread information through current_thread method

#Through current_thread() get thread
current_thread = threading.current_thread()
#Print
print(current_thread)

import threading
import time

def task():
    time.sleep(1)
    thread = threading.current_thread()
    print(thread)

if __name__ == '__main__':
    for i  in range(5):
        sub_thread = threading.Thread(target=task)
        sub_thread.start()
#Execution is out of order

(6) Comparison of, process and thread

Relationship comparison
- Threads are attached to processes. Without processes, there are no threads
- A process provides one thread by default, and a process can create multiple threads

Difference comparison
- The resource cost of creating a process is greater than that of creating a thread
- Process is the basic unit of operating system resource allocation, and thread is the basic unit of cpu scheduling
- Threads cannot be executed alone and must exist in the process
Advantages and disadvantages

(7) , multi process and highly concurrent copy

import os
import threading
def copy_file(file_name,source_dir,dest_dir):
    # 1. Splice source folder path and destination folder path
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    # 2. Open source and destination files
    with open(source_path,"rb") as source_file:   #Read file contents in rb binary mode
        with open(dest_path,"wb") as dest_file:   #wb write file contents in binary mode
    # 3. Loop reading source file to destination path
            while True:
                data =  source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break

if __name__ == '__main__':
    # 1. Define source files and destination folders
    source_dir = r"G:\markdown\python"
    dest_dir = r"D:\python"
    # 2. Create destination folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The target file already exists")
    # 3. Read the file list of the source file
    file_list = os.listdir(source_dir)

    # 4. Traversing the file list to copy
    for file_name in file_list:
        thread =  threading.Thread(target=copy_file,args=(file_name,source_dir,dest_dir))
        thread.start()

Posted by uktrips007 on Wed, 01 Sep 2021 15:54:23 -0700

Programmer Group

python multithreaded programming notes

python multithreaded programming notes

1, Process

(1) Introduction to multitasking

Two ways

1. Concurrent

2. Parallel

(2) Introduction to process

(3) Multi process and multi task

1. Process creation steps

2. The process executes a task with parameters

(4) . obtain the process number

(5) . process considerations

(6) , multitasking copy manager

2, Thread

(1) Introduction to threads

(2) . multithreading to complete multitasking

1. Steps to create a thread

(3) . the thread executes tasks with parameters

(4) End order of, main thread and child thread

(5) Execution order between threads

(6) Comparison of, process and thread

(7) , multi process and highly concurrent copy

Hot Keywords