使用python-multiprocessing与if __name__ =='__main__'相关的谜

Question

I am using multiprocessing to speed up my program and there is an enigma I can not solve. I am using multiprocessing to write a lot of short files (based on a lot of input files) with the function writing_sub_file , and I finally concatenate all these files after the end of all the processes, using the function my_concat . Here are two samples of code. Note that this code is in my main .py file, but the function my_concat is imported from another module. The first one:

if __name__ == '__main__':
    pool = Pool(processes=cpu_count())
    arg_tuple = (work_path, article_dict, cat_len, date_to, time_period, val_matrix)
    jobs = [(group, arg_tuple) for group in store_groups]
    pool.apply_async(writing_sub_file, jobs)
    pool.close()
    pool.join()

my_concat(work_path)

which gives many errors (as many as there are processes in the pool) since It tries to apply my_concat before all my processes are done (I don't give the stack of the error since It is very clear that my_concat function tries to apply before every files have been written by the pool processes).

The second one:

if __name__ == '__main__':
    pool = Pool(processes=cpu_count())
    arg_tuple = (work_path, article_dict, cat_len, date_to, time_period, val_matrix)
    jobs = [(group, arg_tuple) for group in store_groups]
    pool.apply_async(writing_sub_file, jobs)
    pool.close()
    pool.join()
    my_concat(work_path)

which works perfectly.

Can someone explain me the reason?

Answer 1

In the second, my_concat(work_path) is inside the if statement, and is therefore only executed if the script is running as the main script.

In the first, my_concat(work_path) is outside the if statement. When multiprocessing imports the module in a new Python session, it is not imported as __main__ but under its own name. Therefore this statement is run pretty much immediately, in each of your pool's processes, when your module is imported into that process.