[英]Pickling method descriptor objects in python
I am trying to pickle a method_descriptor
. 我想腌制一个method_descriptor
。
Pickling with pickle
or cloudpickle
fails: 用pickle
或cloudpickle
酸洗失败:
Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pickle, cloudpickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: cannot pickle method_descriptor objects
>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
cp.dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.
Importing dill
somehow makes pickle
work, as shown below: 以某种方式导入dill
使得pickle
工作,如下所示:
>>> import dill
>>> pickle.dumps(set.union)
'cdill.dill\n_getattr\np0\n(c__builtin__\nset\np1\nS\'union\'\np2\nS"<method \'union\' of \'set\' objects>"\np3\ntp4\nRp5\n.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> set.union(set([1,2]), set([3]))
set([1, 2, 3])
>>> f(set([1,2]), set([3]))
set([1, 2, 3])
The issue in cloudpickle
remains even after the dill
import: 在dill
导入之后, cloudpickle
的问题仍然存在:
>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
cp.dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.
In my application I rely on cloudpickle
to handle functions with globals. 在我的应用程序中,我依靠cloudpickle
来处理全局变量的函数。 So my question is, how can I get cloudpickle
to work for method_descriptor
objects in Python 2.7? 所以我的问题是,如何在Python 2.7中使用cloudpickle
来处理method_descriptor
对象?
EDIT: I noticed that the same issue occurs in Python 3.3, but is not present in Python 3.5. 编辑:我注意到Python 3.3中出现了同样的问题,但在Python 3.5中并不存在。
I'm the dill
author. 我是dill
作者。 When you do import dill
, it injects the serialization registry from dill
into pickle
(basically, puts all the copy_reg
-type knowledge from dill
into the pickle
registry). 当您import dill
,它会将序列化注册表从dill
注入pickle
(基本上,将所有copy_reg
-type知识从dill
放入pickle
注册表中)。
>>> import pickle
>>> pickle.Pickler.dispatch
{<type 'function'>: <function save_global at 0x105d0c7d0>, <type 'dict'>: <function save_dict at 0x105d0c668>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'type'>: <function save_global at 0x105d0c7d0>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'classobj'>: <function save_global at 0x105d0c7d0>, <type 'builtin_function_or_method'>: <function save_global at 0x105d0c7d0>}
>>> import dill
>>> pickle.Pickler.dispatch
{<class '_pyio.BufferedReader'>: <function save_file at 0x106c8b848>, <class '_pyio.TextIOWrapper'>: <function save_file at 0x106c8b848>, <type 'operator.itemgetter'>: <function save_itemgetter at 0x106c8b578>, <type 'weakproxy'>: <function save_weakproxy at 0x106c8c050>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'file'>: <function save_file at 0x106c8b8c0>, <type 'classmethod'>: <function save_classmethod at 0x106c8c230>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'instancemethod'>: <function save_instancemethod0 at 0x106c8ba28>, <type 'cell'>: <function save_cell at 0x106c8bb18>, <type 'member_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'slice'>: <function save_slice at 0x106c8bc80>, <type 'dict'>: <function save_module_dict at 0x106c8b410>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'code'>: <function save_code at 0x106c8b320>, <type 'type'>: <function save_type at 0x106c8c0c8>, <type 'xrange'>: <function save_singleton at 0x106c8bde8>, <type 'builtin_function_or_method'>: <function save_builtin_method at 0x106c8b9b0>, <type 'classobj'>: <function save_classobj at 0x106c8b488>, <type 'weakref'>: <function save_weakref at 0x106c8bed8>, <type 'getset_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'weakcallableproxy'>: <function save_weakproxy at 0x106c8c050>, <class '_pyio.BufferedRandom'>: <function save_file at 0x106c8b848>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'functools.partial'>: <function save_functor at 0x106c8b7d0>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'function'>: <function save_function at 0x106c8b398>, <type 'thread.lock'>: <function save_lock at 0x106c8b500>, <type 'super'>: <function save_functor at 0x106c8b938>, <type 'staticmethod'>: <function save_classmethod at 0x106c8c230>, <type 'module'>: <function save_module at 0x106c8bf50>, <type 'method_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'operator.attrgetter'>: <function save_attrgetter at 0x106c8b5f0>, <type 'wrapper_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'numpy.ufunc'>: <function save_numpy_ufunc at 0x106c8bcf8>, <type 'method-wrapper'>: <function save_instancemethod at 0x106c8baa0>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'cStringIO.StringI'>: <function save_stringi at 0x106c8b6e0>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <class '_pyio.BufferedWriter'>: <function save_file at 0x106c8b848>, <type 'property'>: <function save_property at 0x106c8c140>, <type 'ellipsis'>: <function save_singleton at 0x106c8bde8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'cStringIO.StringO'>: <function save_stringo at 0x106c8b758>, <type 'NotImplementedType'>: <function save_singleton at 0x106c8bde8>, <type 'dictproxy'>: <function save_dictproxy at 0x106c8bb90>}
cloudpickle
has (slightly) different pickling functions than dill
, and if you are using cloudpickle
, it push it's own serialization functions into the pickle
registry. cloudpickle
有(略)不同于酸洗功能dill
,如果你正在使用cloudpickle
,推它自己的序列化功能到pickle
注册表。 If you want to get cloudpickle
to work for you, you might be able to monkeypatch a solution… essentially install a module within your application that does import dill as cloudpickle
(Nice reference: http://blog.dscpl.com.au/2015/03/safely-applying-monkey-patches-in-python.html )… but that would replace the entire use of cloudpickle
with dill
in your application context. 如果你想让cloudpickle
为你工作,你可能能够monkeypatch一个解决方案...本质上在你的应用程序中安装一个模块, import dill as cloudpickle
(Nice reference: http : import dill as cloudpickle
/03/safely-applying-monkey-patches-in-python.html )...但是这会在你的应用程序环境中用dill
取代cloudpickle
的全部使用。 You could also try a monkeypatch along these lines: 你也可以沿着这些方向尝试一个monkeypatch:
>>> #first import dill, which populates itself into pickle's dispatch
>>> import dill
>>> import pickle
>>> # save the MethodDescriptorType from dill
>>> MethodDescriptorType = type(type.__dict__['mro'])
>>> MethodDescriptorWrapper = pickle.Pickler.dispatch[MethodDescriptorType]
>>> # cloudpickle does the same, so let it update the dispatch table
>>> import cloudpickle
>>> # now, put the saved MethodDescriptorType back in
>>> pickle.Pickler.dispatch[MethodDescriptorWrapperType] = MethodDescriptorWrapper
Note that if you are going to use cloudpickle.dumps
directly, you'd have to overload the registry in cloudpickle
directly by doing the above monkeypatch on cloudpickle.CloudPickler.dispatch
. 请注意,如果你要使用cloudpickle.dumps
直接,你不得不超负荷注册表cloudpickle
直接做上述猴补丁cloudpickle.CloudPickler.dispatch
。
I don't guarantee that it will work, nor do I guarantee that it won't screw up other objects from cloudpickle
(essentially, I haven't tried it), but it's a potential route to replacing the offending cloudpickle
wrapper with the one from dill
. 我不保证它会起作用,也不保证它不会搞砸cloudpickle
其他对象(基本上,我还没有尝试过),但这是用一个替换违规的cloudpickle
包装器的潜在途径。来自dill
。
If you want the short answer, I'd say (at least for this case) use dill
. 如果你想要简短的答案,我会说(至少在这种情况下)使用dill
。 ;) ;)
EDIT with regard to copyreg
: 关于copyreg
编辑 :
Here's what's in dill
: 这是dill
的东西:
def _getattr(objclass, name, repr_str):
# hack to grab the reference directly
try:
attr = repr_str.split("'")[3]
return eval(attr+'.__dict__["'+name+'"]')
except:
attr = getattr(objclass,name)
if name == '__dict__':
attr = attr[name]
return attr
Which is used to register a function with a lower-level reduce function (directly on the pickler instance). 用于注册具有较低级别reduce函数的函数(直接在pickler实例上)。 obj
is the object to pickle. obj
是pickle的对象。
pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__, obj.__repr__()), obj=obj)
I believe this translates to a reduce method (used directly in copyreg.pickle
) like this: 我相信这会转换为reduce方法(直接在copyreg.pickle
),如下所示:
def _reduce_method_descriptor(obj):
return _getattr, (obj.__objclass__, obj.__name__, obj.__repr__())
After much messing around I think I have found a clean answer to this question that works with Python 2.7 and 3.3. 经过多次搞乱之后,我想我已经找到了一个适用于Python 2.7和3.3的问题的简洁答案。 Note that Python 3.5 has no issues to begin with. 请注意,Python 3.5没有问题。
Before I show the results of my findings I want to credit the multiprocessing.forking
module, which was where I got the jist of the code that makes this work. 在我展示我的发现结果之前,我想要归功于multiprocessing.forking
模块,这是我获得使这项工作的代码的要点。
In what follows, I will use set.union
as an example of <class 'method_descriptor'>
. 在下文中,我将使用set.union
作为<class 'method_descriptor'>
的示例。
Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 19 2015, 21:57:25)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.dumps(set.union)
b'\x80\x03cbuiltins\ngetattr\nq\x00cbuiltins\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}
>>>
copyreg
to provide a way for pickle
to work with method_descriptor
Python 3.3:使用copyreg
为pickle
提供了一种方法来使用method_descriptor
Python 3.3.5 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:22:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'method_descriptor'>: attribute lookup builtins.method_descriptor failed
The type of set.union
is method_descriptor
: set.union
的类型是method_descriptor
:
>>> type(set.union)
<class 'method_descriptor'>
We define the reduce function for the method_descriptor and register it with copyreg
: 我们为method_descriptor定义reduce函数并将其注册到copyreg
:
>>> def _reduce_method_descriptor(m):
... return getattr, (m.__objclass__, m.__name__)
...
>>> import copyreg
>>> copyreg.pickle(type(set.union), _reduce_method_descriptor)
Success: 成功:
>>> pickle.dumps(set.union)
b'\x80\x03cbuiltins\ngetattr\nq\x00cbuiltins\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}
And if we now import cloudpickle
the registered pickling function still works: 如果我们现在导入cloudpickle
,注册的酸洗功能仍然有效:
>>> import cloudpickle
>>> cloudpickle.dumps(set.union)
b'\x80\x02c__builtin__\ngetattr\nq\x00c__builtin__\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}
>>>
copy_reg
to provide a way for pickle
to work with method_descriptor
Python 2.7:使用copy_reg
为pickle
提供一种方法来使用method_descriptor
In Python 2.7, the module for registering pickle support functions is called copy_reg
. 在Python 2.7中,用于注册pickle支持函数的模块称为copy_reg
。
Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle method_descriptor objects
The type of set.union
is method_descriptor
: set.union
的类型是method_descriptor
:
>>> type(set.union)
<type 'method_descriptor'>
We define the reduce function for the method_descriptor and register it with copyreg
: 我们为method_descriptor定义reduce函数并将其注册到copyreg
:
>>> def _reduce_method_descriptor(m):
... return getattr, (m.__objclass__, m.__name__)
...
>>> import copy_reg
>>> copy_reg.pickle(type(set.union), _reduce_method_descriptor)
>>> pickle.dumps(set.union)
"c__builtin__\ngetattr\np0\n(c__builtin__\nset\np1\nS'union'\np2\ntp3\nRp4\n."
Success: 成功:
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f(set([1, 2, 3]), set([5]))
And works with cloudpickle
too: 并且也适用于cloudpickle
:
set([1, 2, 3, 5])
>>> import cloudpickle
>>> cloudpickle.dumps(set.union)
'\x80\x02c__builtin__\ngetattr\nq\x00c__builtin__\nset\nq\x01U\x05unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(cloudpickle.dumps(set.union))
>>> f(set([1, 2, 3]), set([5]))
set([1, 2, 3, 5])
>>>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.