[英]Python: can't pickle module objects error
I'm trying to pickle a big class and getting我正在尝试腌制一个大班并获得
TypeError: can't pickle module objects类型错误:无法pickle 模块对象
despite looking around the web, I can't exactly figure out what this means.尽管环顾网络,我无法完全弄清楚这意味着什么。 and I'm not sure which module object
is causing the trouble.而且我不确定是哪个module object
导致了问题。 is there a way to find the culprit?有没有办法找到罪魁祸首? the stack trace doesn't seem to indicate anything.堆栈跟踪似乎没有任何指示。
Python's inability to pickle module objects is the real problem. Python 无法pickle 模块对象才是真正的问题。 Is there a good reason?有充分的理由吗? I don't think so.我不这么认为。 Having module objects unpicklable contributes to the frailty of python as a parallel / asynchronous language.使模块对象不可选择导致 python 作为并行/异步语言的脆弱性。 If you want to pickle module objects, or almost anything in python, then use dill
.如果你想pickle模块对象,或者python中的几乎任何东西,那么使用dill
。
Python 3.2.5 (default, May 19 2013, 14:25:55)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import os
>>> dill.dumps(os)
b'\x80\x03cdill.dill\n_import_module\nq\x00X\x02\x00\x00\x00osq\x01\x85q\x02Rq\x03.'
>>>
>>>
>>> # and for parlor tricks...
>>> class Foo(object):
... x = 100
... def __call__(self, f):
... def bar(y):
... return f(self.x) + y
... return bar
...
>>> @Foo()
... def do_thing(x):
... return x
...
>>> do_thing(3)
103
>>> dill.loads(dill.dumps(do_thing))(3)
103
>>>
Get dill
here: https://github.com/uqfoundation/dill在这里获取dill
: https : //github.com/uqfoundation/dill
I can reproduce the error message this way:我可以通过这种方式重现错误消息:
import cPickle
class Foo(object):
def __init__(self):
self.mod=cPickle
foo=Foo()
with file('/tmp/test.out', 'w') as f:
cPickle.dump(foo, f)
# TypeError: can't pickle module objects
Do you have a class attribute that references a module?你有一个引用模块的类属性吗?
Inspired by wump
's comment: Python: can't pickle module objects error受到wump
评论的启发: Python: can't pickle module objects error
Here is some quick code that helped me find the culprit recursively.这是一些帮助我递归找到罪魁祸首的快速代码。
It checks the object in question to see if it fails pickling.它检查有问题的对象,看它是否无法酸洗。
Then iterates trying to pickle the keys in __dict__
returning the list of only failed picklings .然后迭代尝试pickle __dict__
的键,返回仅失败__dict__
的列表。
import pickle
def pickle_trick(obj, max_depth=10):
output = {}
if max_depth <= 0:
return output
try:
pickle.dumps(obj)
except (pickle.PicklingError, TypeError) as e:
failing_children = []
if hasattr(obj, "__dict__"):
for k, v in obj.__dict__.items():
result = pickle_trick(v, max_depth=max_depth - 1)
if result:
failing_children.append(result)
output = {
"fail": obj,
"err": e,
"depth": max_depth,
"failing_children": failing_children
}
return output
import redis
import pickle
from pprint import pformat as pf
def pickle_trick(obj, max_depth=10):
output = {}
if max_depth <= 0:
return output
try:
pickle.dumps(obj)
except (pickle.PicklingError, TypeError) as e:
failing_children = []
if hasattr(obj, "__dict__"):
for k, v in obj.__dict__.items():
result = pickle_trick(v, max_depth=max_depth - 1)
if result:
failing_children.append(result)
output = {
"fail": obj,
"err": e,
"depth": max_depth,
"failing_children": failing_children
}
return output
if __name__ == "__main__":
r = redis.Redis()
print(pf(pickle_trick(r)))
$ python3 pickle-trick.py
{'depth': 10,
'err': TypeError("can't pickle _thread.lock objects"),
'fail': Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>,
'failing_children': [{'depth': 9,
'err': TypeError("can't pickle _thread.lock objects"),
'fail': ConnectionPool<Connection<host=localhost,port=6379,db=0>>,
'failing_children': [{'depth': 8,
'err': TypeError("can't pickle _thread.lock objects"),
'fail': <unlocked _thread.lock object at 0x10bb58300>,
'failing_children': []},
{'depth': 8,
'err': TypeError("can't pickle _thread.RLock objects"),
'fail': <unlocked _thread.RLock object owner=0 count=0 at 0x10bb58150>,
'failing_children': []}]},
{'depth': 9,
'err': PicklingError("Can't pickle <function Redis.<lambda> at 0x10c1e8710>: attribute lookup Redis.<lambda> on redis.client failed"),
'fail': {'ACL CAT': <function Redis.<lambda> at 0x10c1e89e0>,
'ACL DELUSER': <class 'int'>,
0x10c1e8170>,
.........
'ZSCORE': <function float_or_none at 0x10c1e5d40>},
'failing_children': []}]}
In my case, creating an instance of Redis
that I saved as an attribute of an object broke pickling.在我的例子中,创建一个我保存为对象属性的Redis
实例破坏了酸洗。
When you create an instance of Redis
it also creates a connection_pool
of Threads
and the thread locks can not be pickled.当您创建Redis
实例时,它还会创建一个Threads
的connection_pool
,并且线程锁不能被酸洗。
I had to create and clean up Redis
within the multiprocessing.Process
before it was pickled.在腌制之前,我必须在multiprocessing.Process
创建和清理Redis
。
In my case, the class that I was trying to pickle, must be able to pickle.就我而言,我试图腌制的班级必须能够腌制。 So I added a unit test that creates an instance of the class and pickles it.所以我添加了一个单元测试来创建类的一个实例并对其进行腌制。 That way if anyone modifies the class so it can't be pickled, therefore breaking it's ability to be used in multiprocessing (and pyspark), we will detect that regression and know straight away.这样,如果有人修改了该类,使其无法被腌制,从而破坏了它在多处理(和 pyspark)中使用的能力,我们将检测到该回归并立即知道。
def test_can_pickle():
# Given
obj = MyClassThatMustPickle()
# When / Then
pkl = pickle.dumps(obj)
# This test will throw an error if it is no longer pickling correctly
According to the documentation:根据文档:
What can be pickled and unpickled?什么可以腌制和不腌制?
The following types can be pickled:可以腌制以下类型:
None, True, and False无、真和假
integers, floating point numbers, complex numbers整数、浮点数、复数
strings, bytes, bytearrays字符串、字节、字节数组
tuples, lists, sets, and dictionaries containing only picklable objects仅包含可picklable 对象的元组、列表、集合和字典
functions defined at the top level of a module (using def, not lambda)在模块顶层定义的函数(使用 def,而不是 lambda)
built-in functions defined at the top level of a module定义在模块顶层的内置函数
classes that are defined at the top level of a module在模块顶层定义的类
instances of such classes whose
__dict__
or the result of calling__getstate__()
is picklable (see section Pickling Class Instances for details).__dict__
或调用__getstate__()
的结果是可__getstate__()
的此类类的实例(有关详细信息,请参阅酸洗类实例部分)。
As you can see, modules are not part of this list.如您所见,模块不在此列表中。 Note, that this is also true when using deepcopy
and not only for the pickle
module, as stated in the documentation of deepcopy
:请注意,如deepcopy
的文档中所述,这在使用deepcopy
时也是正确的,而不仅仅是用于pickle
模块:
This module does not copy types like module, method, stack trace, stack frame, file, socket, window, array, or any similar types.此模块不复制模块、方法、堆栈跟踪、堆栈帧、文件、套接字、窗口、数组或任何类似类型等类型。 It does “copy” functions and classes (shallow and deeply), by returning the original object unchanged;它通过返回原始对象不变来“复制”函数和类(浅的和深的); this is compatible with the way these are treated by the pickle module.这与 pickle 模块处理这些的方式兼容。
A possible workaround is using the @property
decorator instead of an attribute.一种可能的解决方法是使用@property
装饰器而不是属性。 For example, this should work:例如,这应该有效:
import numpy as np
import pickle
class Foo():
@property
def module(self):
return np
foo = Foo()
with open('test.out', 'wb') as f:
pickle.dump(foo, f)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.