Python：如何使用多重处理来加快对类实例的操作？

Question

for example:

instances = [ClassName() for i in range(no_inst)]
for data in dataset:
    for inst in instances:
        inst.dosomething(data)

Note: The instances do not modify the data, so there is no need of communication between the instances.

Answer 1

The easiest way that I know of is to use multiprocessing.Pool.map the same way you would use the map builtin. You just need to write a function to do the task that you want to complete and it handles all of the parallelism:

import multiprocessing as mp

instances = [ClassName() for i in range(no_inst)]

def dosomething(inst):
    for data in dataset:
        inst.dosomething(data)

num_procs = 4 #however many processors you'll use
pool = mp.Pool(num_procs)
pool.map(dosomething,instances)

there are a few gotchas -- lambda functions won't work here since they aren't pickleable for instance. This also returns a list of None (since that is what dosomething returns) -- normally I don't like using list-comprehensions or map for side-effects, but I relax my stance on that when it comes to multiprocessing simply because it is so easy .

Answer 2

You can use the threading interface to create a Thread class that fulfills the dosomething(data) . You can start multiple thread for each loop, calling run() to start them. It's recommended to then put them into a list so that you can check their status until all have completed.