“多处理”会自动关闭完成的子进程吗？

Question

I used the multiprocessing lib to create multi-thread to process a list of files(20+ files).

When I run the py file, I set the pool number as 4. But in cmd, it showed there are over 10 processes. And most of them have been running for a long time. Because it's large file and takes long time to process so I'm not sure if the process is hanging or still executing.

So my question is:

if it's executing, how to set the process number as exactly 4?

if it's hanging, it means child process will not shut down after finished. Can I set it automatically shutting down after finished?

from multiprocessing import Pool
poolNum = int(sys.argv[1])
pool = Pool(poolNum)
pool.map(processFunc, fileList)

Answer 1

It won't, not until the Pool is close -ed or terminate -ed (IIRC Pool s at least at present have a reference cycle involved, so even when the last live reference to the Pool goes away, the Pool is not deterministically collected, even on CPython, which uses reference counting and normally has deterministic behavior).

Since you're using map , your work is definitely done when map returns, so the simplest solution is just to use a with statement for guaranteed termination:

from multiprocessing import Pool

def main():
    poolNum = int(sys.argv[1])

    with Pool(poolNum) as pool:  # Pool created
        pool.map(processFunc, fileList)
    # terminate has been called, all workers will be killed

# Adding main guard so this code is valid on Windows and anywhere else which
# doesn't use forking for whatever reason
if __name__ == '__main__':
    main()

As I commented, I used a main function with the standard guard against being invoked on import , as Windows simulates forking by reimporting the main module (but not naming it __main__ ); without the guard, you can end up with the child process creating new processes automatically, which is problematic.

Side-note: If you are dispatching a bunch of tasks but not waiting on them immediately (so you don't want to terminate the pool anywhere near when you create it, but want to ensure the workers are cleaned up promptly), you can still use context management to help out. Just use contextlib.closing to close the pool once all the tasks are dispatched; you must dispatch all the tasks before the end of the with block, but you can retrieve the results later, and when all results are computed, the child processes will close. For example:

from contextlib import closing
from multiprocessing import Pool

def main():
    poolNum = int(sys.argv[1])

    with closing(Pool(poolNum)) as pool:  # Pool created
        results = pool.imap_unordered(processFunc, fileList)
    # close has been called, so no new work can be submitted,
    # and when all outstanding tasks complete, the workers will exit
    # immediately/cleanly

    for res in results:
        # Can still retrieve results even after pool is closed

# Adding main guard so this code is valid on Windows and anywhere else which
# doesn't use forking for whatever reason
if __name__ == '__main__':
    main()