在循环的每次迭代中运行一个函数作为 python 中的一个新进程

Question

I have this:

from multiprocessing import Pool

pool = Pool(processes=4)

def createResults(uniqPath):
    *(there is some code here that populates a list - among other things)*

for uniqPath in uniqPaths:
    pool.map(createResults, uniqPath)

pool.close()
pool.join()

I don't know if it's possible, but can I run the createResults function that gets called in that loop as a new process for each iteration?

I'm populating a list using a 4 million line file and it's taking 24+ hours to run. (Obviously the code above does not work)

Thanks!

Answer 1

Instead of:

for uniqPath in uniqPaths:
    pool.map(createResults, uniqPath)

Do this:

pool.map(createResults, uniqPaths)

You must use map on the iterable itself in order to run in a concurrent fashion.
Keep in mind though - Populating a list means the list won't be shared between the processes, and if it does using Array() , make sure it's process-safe.