在Python中丢失排队的任务apply_async

Question

I am attempting to write a wrapper that iterates through running another program with different input files. The program (over which I have no control, but need to use) needs to be run out of the same directory as the input file(s). So far my method is to use OS-module to change/create directory structure, use apply-async to run the program given a sub-directory, and each child in apply-async changes directory, creates the file, and the first 8 processes run successfully (i have 8 virtual cores)

However, I am queueing up to 100 of these processes (they run a simulation which takes a few minutes, I'm looking to optimize). I use "call" on the outside executable I am running. I thought everything was going great, but then after the 8th simulation runs, everything stops, I check 0 processes are running. It is as if the queue forgot about the other processes.

What can I do to fix this? I know my RAM only goes up about 300 MB out of 8GB.

Do I need to look into implementing some sort of queue myself that waits for the exit code of the simulation executable?

Thank you in advance.

Answer 1

Maybe better than nothing. This shows a correct way to use apply_async() , and demonstrates that - yup - there's no problem creating many more tasks than processes. I'd tell you how this differs from what you're doing, but I have no idea what you're doing ;-)

import multiprocessing as mp

def work(i):
    from time import sleep
    sleep(2.0 if i & 1 else 1.0)
    return i*i

if __name__ == "__main__":
    pool = mp.Pool(4)
    results = [pool.apply_async(work, (i,)) for i in range(100)]
    results = [r.get() for r in results]
    print len(results), results
    pool.close()
    pool.join()