在 Python 中使用 Pool 类进行多处理会导致 Pickling 错误

Question

I am trying to run a simple multiprocessing example in python3.6 in a zeppelin notebook(in windows) but I am not able to execute it. Below is the code that i used:

def sqrt(x):
    return x**0.5

numbers = [i for i in range(1000000)]
with Pool() as pool:
    sqrt_ls = pool.map(sqrt, numbers)

After running this code I am getting the following error:

Traceback (most recent call last):
  File "/tmp/zeppelin_python-3196160128578820301.py", line 315, in <module>
    exec(code, _zcUserQueryNameSpace)
  File "<stdin>", line 6, in <module>
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function sqrt at 0x7f6f84f1a620>: attribute lookup sqrt on __main__ failed

I am not sure if its just me who is facing the issue. As i have seen so many articles where people can run the code easily. If you know the solution please help

Thanks

Answer 1

From the multiprocessing documentation :

Note: Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the Pool examples will not work in the interactive interpreter.

Notebooks are running Python interactive interpreters behind the scene, so it's probably why you get this error. You can try to run your code from within a if __name__ == '__main__': statement.

Answer 2

A Zeppelin notebook does not emulate a normal module well enough to support the pickling that is used to identify the correct operation to another process. You can put all the functions you want to call into a proper module that you import in the usual fashion.