I am new to multiprocessing in python and have run into some issues with Pool: OS: Mac Monterey M1 chip Python 3.9.12
in module.py: I have tried
def foo(x)
...
return y
pool = mp.Pool(8)
results = pool.map_async(foo, args)
also tried pathos:
def foo(x)
...
return y
pool = ProcessPool(8)
results = pool.amap(foo, args)
in main script:
import module
def main():
test = module.foo(x)
if __name__ == "__main__":
main()
I also installed the package to local using setup.py.
current error message including: for multiprocessing
AttributeError: Can't pickle local object 'search.<locals>.foo
and for pathos
TypeError: no default __reduce__ due to non-trivial __cinit__
PS: I am new to stackoverflow. I am trying my best to state the problems. I am not sure what information might need to solve the problem. I have been updating the problem every time people asking for new information as soon as I can. So please be polite.
I'm the author of dill
, pathos
, and multiprocess
. It looks like you have an object that won't serialize. Your question doesn't provide enough information for me to give you a solution that I know will work -- but I can give you some things to try.
dill
: Python 3.7.15 (default, Oct 12 2022, 04:11:53)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.settings['recurse'] = True
The above will change how objects in the global namespace are handled, so if the function you are interested in passing serializes ok, but there's a unserializable object in the global dict... then the change to the above serialization variant may avoid the "bad" object. You could also identify the object that's causing the failure, and delete it from your current namespace. However, if it's needed by the function you are interested in, then you alternately need to refactor the code.
__reduce__
method, or similar (eg set/get state methods). See the__reduce__
docs in pickle
. The error you are seeing from pathos
tells me that you are running in to an object that is written in C (it has a __cinit__
) and thus needs some additional methods to tell pickle
or dill
how to save its state. The problem you might have here is that you didn't create the object in question, and it comes from some module you are importing. This leads to case #3, but without details on your code, I can't say much more.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.