Python concurrency with imported modules

Question

Why does importing local modules cause Concurrent.futures.ProcessPoolExecutor to throw a BrokenProcessPool exception?

When I import a local module, I get a BrokenProcessPool exception when I run the file. I have tried commenting out everything in that module and I get the same result. I have also tried other files/modules with the same result. However, if I comment out the import statement, or place it inside the main() function, it works without terminating a process and raising the exception. I have tried the same thing with other local modules and I get the same result. Why is this happening and what can I do to avoid the exception?

I am trying to use concurrent.futures with ProcessPoolExecutor. I based my code example on the top answer for this question: Parallelize apply after pandas groupby

Here is my version:

import pandas as pd
import numpy as np
import time
from concurrent.futures import ProcessPoolExecutor, as_completed
import analysis_helper # a local module

print(__name__)
nrows = 15000
np.random.seed(1980)
df = pd.DataFrame({'a': np.random.permutation(np.arange(nrows))})

def f1(group):
    time.sleep(0.0001)
    return group

def main():
    with ProcessPoolExecutor(12) as ppe:
        futures = []
        results = []

        for name, group in df.groupby('a'):
            p = ppe.submit(f1, group)
            futures.append(p)

        for future in as_completed(futures):
            r = future.result()
            results.append(r)

        df_output = pd.concat(results)
        print(df_output)

if __name__ == '__main__':
    main()

Results with analysis_helper removed:

runfile('C:/dev/.../test_parallelizer_pandas.py', wdir='C:/dev/...')
__main__
           a
1255    1733
3372   11015
5318    4571
7076   14510
10545  10749
3340     483
11844   3736
3681   14509
2222    1041
3640   11014
4288    7852
12257   1040
2101   11034
14938   3065
8449    1842
7231   10746
7509    4353
4898    3797
2941     866
7497   14520
8302   11013
13882   9924
12007   1042
1567   10747
13135   7856
7742     485
13709  12571
1946   11012
5634    7848
7044    4354
     ...
3441   14213
179    14361
6723   12134
7528    5905
9273   12420
9916    3614
134    10166
11654   5854
11848  12133
14055   4278
6100   14360
726    14981
13139  14982
12552  14983
5393   14984
6927   14986
8108   14985
12665  14987
8587   14988
11437  14989
4191   14990
6877   14991
4997   14994
13527  14995
9477   14993
2930   14996
5456   14992
781    14997
3287   14998
13386  14999

[15000 rows x 1 columns]

Results with analysis_helper:

runfile('C:/dev/.../test_parallelizer_pandas.py', wdir='C:/dev/...')
__main__
Traceback (most recent call last):

  File "<ipython-input-7-7d6a88ec5a87>", line 1, in <module>
    runfile('C:/dev/.../test_parallelizer_pandas.py', wdir='C:/dev/...')

  File "C:\Users\david\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "C:\Users\david\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/dev/.../test_parallelizer_pandas.py", line 42, in <module>
    main()

  File "C:/dev/.../test_parallelizer_pandas.py", line 35, in main
    r = future.result()

  File "C:\Users\david\Anaconda3\lib\concurrent\futures\_base.py", line 425, in result
    return self.__get_result()

  File "C:\Users\david\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Note: this only happens with ProcessPoolExecutor, not ThreadPoolExecutor.

Answer 1

If you are using macOS, defining one of the below environment variables could work as a temporary workaround.

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES  

Read more about the issue here

NO_PROXY='*'

Read more about the issue here

The second option worked for me on macOS 10.14.1 with Python 3.6.0