python multithreaded programming notes

Keywords: Python

;

python multithreaded programming notes

Information: Video tutorial

Time: August 31, 2021

1, Process

(1) Introduction to multitasking

Multitasking: execute multiple tasks at the same time

Advantages of multitasking: the biggest advantage is to make full use of cpu resources and improve the execution efficiency of programs

Two ways

1. Concurrent

2. Parallel

(2) Introduction to process

Process is the smallest unit of resource allocation. It is the basic unit of resource allocation and scheduling of the operating system (eg: the executing program is a process)

(3) Multi process and multi task

1. Process creation steps

  • Import process package

    • import multiprocessing
      
  • Creating process objects through process classes

    • Process object = multiprocressing.Process(target=Task name)
      

  • Start the process and execute the task

    • Process object.start()
      
#1. Import package
import multiprocessing
import time
def sing():
    for i in range(3):
        print("Singing")
        time.sleep(0.3)
def dance():
    for j in range(3):
        print("Dance")
        time.sleep(0.3)


if __name__ == '__main__':
    #2. Create process
    process_sing = multiprocessing.Process(target=sing)
    process_dance = multiprocessing.Process(target=dance)
    #3. Start process
    process_dance.start()
    process_sing.start()

2. The process executes a task with parameters

Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name

#Import package
import multiprocessing
import time
def sing(num,name):
    print("%s sing"%name)
    for i in range(num):
        print("sing%d......"%i)
        time.sleep(0.3)
def dance(num):
    for j in range(num):
        print("dance%d......"%j)
        time.sleep(0.3)


if __name__ == '__main__':
    #Create process
    # target: task name
    #args: pass parameters to the function as tuples
    #kwargs: pass parameters to functions in dictionary mode
    #Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name
    process_sing = multiprocessing.Process(target=sing,args=(3,"zqq"))
    process_dance = multiprocessing.Process(target=dance,kwargs={"num":4})
    #Start process
    process_dance.start()
    process_sing.start()

(4) . obtain the process number

1. Two ways to get the process number

  • Get current process number

    os.getpid()

  • Gets the current parent process number

    os.getppid()

2. Method of use

  • Guide Package

    import os
    

    #Import package
    import multiprocessing
    import time
    import os
    #sing
    def sing(num,name):
        print("Singing process pid",os.getpid())  #Current process number
        print("Main process of singing pid",os.getppid())  #Parent process number
        print("%s sing"%name)
        for i in range(num):
            print("sing%d......"%i)
            time.sleep(0.3)
    #dance
    def dance(num):
        print("The process of dancing pid",os.getpid())
        print("Main process of dancing pid",os.getppid())
        for j in range(num):
            print("dance%d......"%j)
            time.sleep(0.3)
    
    
    if __name__ == '__main__':
        print("Of the main process pid:%d\n"%os.getpid(),"------"*10)
        #Create process
        # target: task name
        #args: pass parameters to the function as tuples
        #kwargs: pass parameters to functions in dictionary mode
        #Note: for args parameter transmission, pay attention to the sequence of parameters. For kwargs parameter transmission, pay attention to the consistency between key name and parameter name
        process_sing = multiprocessing.Process(target=sing,args=(3,"zqq"))
        process_dance = multiprocessing.Process(target=dance,kwargs={"num":4})
        #Start process
        process_dance.start()
        process_sing.start()
    

(5) . process considerations

1. The main process will wait for the execution of all child processes to end

eg:

import multiprocessing
import time
def work():
    for i in range(10):
        print("At work")
        time.sleep(0.2)
if __name__ == '__main__':
    work_process = multiprocessing.Process(target=work())
   # work_process.daemon = True
    work_process.start()

    time.sleep(1)
    print("End of main process")

2. Set daemon (main process)

Process object. daemon = True

import multiprocessing
import time
def work():
    for i in range(10):
        print("At work")
        time.sleep(0.2)
if __name__ == '__main__':
    work_process = multiprocessing.Process(target=work())
    #Start the main thread guard, the main thread ends, and other threads also end 
    work_process.daemon = True
    work_process.start()
    time.sleep(1)
    print("End of main process")

(6) , multitasking copy manager

1. Demand analysis

  • Whether the target folder exists. If it exists, it will not be created. If it does not exist, it will be created
  • Traverse all files in the source file and copy to the destination folder
  • Multi task is realized by process to complete high concurrent copy
import os
import multiprocessing
def copy_file(file_name,source_dir,dest_dir):
    # 1. Splice source folder path and destination folder path
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    # 2. Open source and destination files
    with open(source_path,"rb") as source_file:   #Read file contents in rb binary mode
        with open(dest_path,"wb") as dest_file:   #wb write file contents in binary mode
    # 3. Loop reading source file to destination path
            while True:
                data =  source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break

if __name__ == '__main__':
    # 1. Define source files and destination folders
    source_dir = r"G:\markdown\python"
    dest_dir = r"D:\python"
    # 2. Create destination folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The target file already exists")
    # 3. Read the file list of the source file
    file_list = os.listdir(source_dir)

    # 4. Traversing the file list to copy
    for file_name in file_list:
        # copy_file(file_list, source_dir, dest_dir)
        # 5. Multi task copy using multi process
        process = multiprocessing.Process(target=copy_file,args=(file_name,source_dir,dest_dir))
        process.start()

2, Thread

(1) Introduction to threads

(2) . multithreading to complete multitasking

1. Steps to create a thread

  • Import thread module

    import threading
    
  • Create thread object

    Thread object = threading.Thread(target = Task name)
    
  • Start the thread to execute the task

    Thread object.start()
    
import time
import threading
def sing():
    for i in range(3):
        print("sing:",i)
        time.sleep(1)  #Rest for 1s
def dance():
    for j in range(3):
        print("dance:",j)
        time.sleep(1)

if __name__ == '__main__':
    #sing()
    #dance()
    #Create thread
    sing_thread = threading.Thread(target = sing)
    dance_thread = threading.Thread(target = dance)
    #Open thread
    sing_thread.start()
    dance_thread.start()

(3) . the thread executes tasks with parameters

import time
import threading
def sing(num):
    for i in range(num):
        print("sing:",i)
        time.sleep(1)  #Rest for 1s
def dance(num):
    for j in range(3):
        print("dance:",j)
        time.sleep(1)

if __name__ == '__main__':
    #Create thread
    #args: pay attention to the order of parameters when passing parameters to the function. Parameters are tuples
    #kwargs: pass in the form of dictionary. Note: the parameter name must be consistent with
    sing_thread = threading.Thread(target = sing,args=(3,))
    dance_thread = threading.Thread(target = dance,kwargs={"num":3})
    #Open thread
    sing_thread.start()
    dance_thread.start()

(4) End order of, main thread and child thread

The main thread will wait for the execution of all child threads to end before the main thread ends

There are two ways to set up daemons:

  • When creating a thread, add the daemon parameter and set it to True

    threading.Thread(target=Task name,daemon=True)
    
  • Using damon

    Thread object.demon = True
    
import threading
import time
def work():
    for i in range(10):
        print("At work",i)
        time.sleep(0.5)
if __name__ == '__main__':
    work_thread = threading.Thread(target=work,daemon=True)
    #Method 1: set daemon thread
    # work_thread.daemon = True
    work_thread.start()
    time.sleep(4)
    print("Main thread end")

(5) Execution order between threads

The order of execution between threads is out of order

  • Get the current thread information through current_thread method

    #Through current_thread() get thread
    current_thread = threading.current_thread()
    #Print
    print(current_thread)
    
    import threading
    import time
    
    def task():
        time.sleep(1)
        thread = threading.current_thread()
        print(thread)
    
    if __name__ == '__main__':
        for i  in range(5):
            sub_thread = threading.Thread(target=task)
            sub_thread.start()
    #Execution is out of order
    

(6) Comparison of, process and thread

  1. Relationship comparison
    • Threads are attached to processes. Without processes, there are no threads
    • A process provides one thread by default, and a process can create multiple threads

  1. Difference comparison

    • The resource cost of creating a process is greater than that of creating a thread
    • Process is the basic unit of operating system resource allocation, and thread is the basic unit of cpu scheduling
    • Threads cannot be executed alone and must exist in the process
  2. Advantages and disadvantages

(7) , multi process and highly concurrent copy

import os
import threading
def copy_file(file_name,source_dir,dest_dir):
    # 1. Splice source folder path and destination folder path
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    # 2. Open source and destination files
    with open(source_path,"rb") as source_file:   #Read file contents in rb binary mode
        with open(dest_path,"wb") as dest_file:   #wb write file contents in binary mode
    # 3. Loop reading source file to destination path
            while True:
                data =  source_file.read(1024)
                if data:
                    dest_file.write(data)
                else:
                    break

if __name__ == '__main__':
    # 1. Define source files and destination folders
    source_dir = r"G:\markdown\python"
    dest_dir = r"D:\python"
    # 2. Create destination folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The target file already exists")
    # 3. Read the file list of the source file
    file_list = os.listdir(source_dir)

    # 4. Traversing the file list to copy
    for file_name in file_list:
        thread =  threading.Thread(target=copy_file,args=(file_name,source_dir,dest_dir))
        thread.start()

Posted by uktrips007 on Wed, 01 Sep 2021 15:54:23 -0700