[英]Cant Pickle memoized class instance
Here is the code I am using 这是我正在使用的代码
import funcy
@funcy.memoize
class mystery(object):
def __init__(self, num):
self.num = num
feat = mystery(1)
with open('num.pickle', 'wb') as f:
pickle.dump(feat,f)
Which is giving me the following error: 这给了我以下错误:
PicklingError: Can't pickle <class '__main__.mystery'>: it's not the
same object as __main__.mystery
I am hoping to 1) understand why this is happening, and 2) find a solution that allows me to pickle the object (without removing the memoization). 我希望1)理解为什么会这样,2)找到一个允许我挑选对象的解决方案(不删除memoization)。 Ideally the solution would not change the call to pickle.
理想情况下,解决方案不会改变对pickle的调用。
Running python 3.6 with funcy==1.10 用funcy运行python 3.6 == 1.10
The problem is that you've applied a decorator designed for functions to a class. 问题是你已经为一个类应用了一个为函数设计的装饰器。 The result is not a class, but a function that wraps up a call to the class.
结果不是一个类,而是一个包装对类的调用的函数。 This causes a number of problems (eg, as pointed out by Aran-Fey in the comments, you can't
isinstance(feat, mystery)
, because mystery
). 这导致了许多问题(例如,正如Aran-Fey在评论中指出的那样,你不可能是
isinstance(feat, mystery)
,因为mystery
)。
But the particular problem you care about is that you can't pickle instances of inaccessible classes. 但是你关心的特殊问题是你不能挑选无法访问的类的实例。
In fact, that's basically what the error message is telling you: 事实上,这基本上是错误消息告诉你的:
PicklingError: Can't pickle <class '__main__.mystery'>: it's not the
same object as __main__.mystery
Your feat
thinks its type is __main__.mystery
, but that isn't a type at all, it's the function returned by the decorator that wraps that type. 你的
feat
认为它的类型是__main__.mystery
,但这根本不是一个类型,它是包装该类型的装饰器返回的函数。
The easy way to fix this would be to find a class decorator meant that does what you want. 解决这个问题的简单方法是找到一个类装饰器,它可以满足您的需求。 It might be called something like
flyweight
instead of memoize
, but I'm sure plenty of examples exist. 它可能被称为
flyweight
而不是memoize
,但我确信存在大量的例子。
But you can build a flyweight class by just memoizing the constructor, instead of memoizing the class: 但是你可以通过记住构造函数来构建一个flyweight类,而不是记住这个类:
class mystery:
@funcy.memoize
def __new__(cls, num):
return super().__new__(cls)
def __init__(self, num):
self.num = num
… although you probably want to move the initialization into the constructor in that case. ...虽然您可能希望在这种情况下将初始化移动到构造函数中。 Otherwise, calling
mystery(1)
and then mystery(1)
will return the same object as before, but also reinitialize it with self.num = 1
, which is at best wasteful, and at worst incorrect. 否则,调用
mystery(1)
然后mystery(1)
将作为前返回相同的对象,但也有重新初始化self.num = 1
,这充其量是浪费的,在最坏的情况不正确。 So: 所以:
class mystery:
@funcy.memoize
def __new__(cls, num):
self = super().__new__(cls)
self.num = num
return self
And now: 现在:
>>> feat = mystery(1)
>>> feat
<__main__.mystery at 0x10eeb1278>
>>> mystery(2)
<__main__.mystery at 0x10eeb2c18>
>>> mystery(1)
<__main__.mystery at 0x10eeb1278>
And, because the type of feat
is now a class that's accessible under the module-global name mystery
, pickle
will have no problem with it at all: 而且,因为
feat
的类型现在是一个可以在模块 - 全局名称mystery
下访问的类,所以pickle
将完全没有问题:
>>> pickle.dumps(feat)
b'\x80\x03c__main__\nmystery\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00numq\x03K\x01sb.'
You do still want to think about how this class should play with pickling. 你还是要考虑这个类应该如何与酸洗玩。 In particular, do you want unpickling to go through the cache?
特别是,你想要unpickling通过缓存吗? By default, it doesn't:
默认情况下,它不会:
>>> pickle.loads(pickle.dumps(feat)) is feat
False
What's happening is that it's using the default __reduce_ex__
for pickling, which defaults to doing the equivalent of (only slightly oversimplified): 发生的事情是它使用默认的
__reduce_ex__
进行酸洗,默认情况下相当于(仅略微过度简化):
result = object.__new__(__main__.mystery)
result.__dict__.update({'num': 1})
If you want it to go through the cache, the simplest solution is this: 如果您希望它通过缓存,最简单的解决方案是:
class mystery:
@funcy.memoize
def __new__(cls, num):
self = super().__new__(cls)
self.num = num
return self
def __reduce__(self):
return (type(self), (self.num,))
If you plan to do this a lot, you might think of writing your own class decorator: 如果你计划这么做,你可能会想到编写自己的类装饰器:
def memoclass(cls):
@funcy.memoize
def __new__(cls, *args, **kwargs):
return super(cls, cls).__new__(cls)
cls.__new__ = __new__
return cls
But this: 但是这个:
__init__
(or, at least, that have an idempotent and fast __init__
that's harmless to call repeatedly), __init__
(或者至少具有幂等且快速的__init__
,无法重复调用), So, I think you're better off being explicit and just memoizing the __new__
method, or writing (or finding) something a lot fancier that does the introspection needed to make memoizing a class this way fully general. 所以,我认为你最好是明确的,只是
__new__
方法,或者写一些(或找到)更多更好的东西,这样做的内省需要以这种方式完全记住一个类。 (Or, alternatively, maybe write one that only works with some restricted set of classes—eg, a @memodataclass
that's just like @dataclass
but with a memoized constructor would be a lot easier than a fully general @memoclass
.) (或者,也可以写一个仅适用于某些受限制的类集合的类 - 例如
@memodataclass
就像@dataclass
但是使用memoized构造函数将比完全通用的@memoclass
容易得多。)
Another approach is 另一种方法是
class _mystery(object):
def __init__(self, num):
self.num = num
@funcy.memoize
def mystery(num):
return _mystery(num)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.