Last article Speaking of Python multi-threaded multi-processing module in the windows platform and Unix platform system in the underlying implementation methods are different, so the efficiency and memory footprint will be different, this paper continues to record how to achieve multi-threaded parallel in different platforms.

Contents

1. Basic multi-process call functions and modules.
2. Call multiple processes in class member functions.
3. Some problems encountered in calling multiple processes in class member functions.

How to call multiprocess functions

Take multiprocessing.Pool as an example:

file1.py

def target_fun():
	return 1
	

main function:

import multiprocessing
from file1 import target_fun

if __name__ == '__main__':
	pool = multiprocessing.Pool(N)
	# N is the number of pools, and if there is a vacancy, the number of pools is the number of CPUs
	res = pool.starmap(target_fun, [[] for _ in range(10)])
	# The advantage of pool.starmap over pool.map is that it can input 0-arbitrary input parameters, while pool.map has only one parameter. The latter parameters control how many queues enter the pool's subprocesses.
	pool.close()
	# The pool closes, no longer accepts new processes, and exits automatically after the existing processes have finished.
	pool.join()
	# The parent process is waiting for the child process to finish

Call multiprocess. Pool () in the class's member functions

class_define.py:

class MC:
	def __init__(self):
		self.nstep = 30000
		# Total calculation 30,000 steps
		
	def MCstep(self):
		self.nstep = self.nstep//10
		# Each subprocess counts 3000 steps
		do_chunk()
		# Do things
		return res
		# Return result
		
	def MCseries(self):
		pool = multiprocessing.Pool(10)
		res_list = pool.starmap(self.MCstep, [[] for _ in range(10)])
		# Create 10 subprocesses, return and collect results.
		pool.close()
		pool.join()
	

main_fun.py

from class_define import MC

mc_test = MC()
# Instantiated class

if __name__ == '__main__':
# Only processes that call this function directly are allowed to perform the following operations to prevent subprocesses from iterating through circular calls
	mc_test.MCseries()
	# Calling member functions of instances

Possible error reporting

1. TypeError: cannot serialize '_io.TextIOWrapper' object

Cause: The MC class contains member variables that cannot be picked, such as Files object. As a result, when a child process is created in the member function of an instance, it is not possible to pick all the members of the current instance and copy them to the child process.

Solution: Delete Files object before pickle and recreate the object in the sub-process. Code:

class_define2.py:

class MC:
	def __init__(self):
		self.nstep = 30000
		# Total calculation 30,000 steps
		self.f = open('haha.txt', 'w')
		
	def __getstate__(self):
		"""Delete from the dictionary self.f"""
		state = self.__dict__.copy()
		del state['f']
		return state
		
	def __setstate__(self, state):
		"""Called while unpickling."""
		self.__dict__.update(state)
	
	def MCstep(self):
		self.f = open('haha.txt', 'w')
		"""Recreate file objects"""
		self.nstep = self.nstep//10
		# Each subprocess counts 3000 steps
		do_chunk()
		# Do things
		return res
		# Return result
		
	def MCseries(self):
		pool = multiprocessing.Pool(10)
		res_list = pool.starmap(self.MCstep, [[] for _ in range(10)])
		# Create 10 subprocesses, return and collect results.
		pool.close()
		pool.join()
	

problem Reference resources, File，pickleDetailed explanation