[英]pickling lru_cached function on object
As part of parallellizing some existing code (with multiprocessing), I run into the situation that something similar to the class below needs to be pickled.作为并行化某些现有代码(使用多处理)的一部分,我遇到了类似于下面的 class 需要腌制的情况。
Starting from:从...开始:
import pickle
from functools import lru_cache
class Test:
def __init__(self):
self.func = lru_cache(maxsize=None)(self._inner_func)
def _inner_func(self, x):
# In reality this will be slow-running
return x
calling打电话
t = Test()
pickle.dumps(t)
returns返回
_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000190454A7AC8>: it's not the same object as __main__.Test._inner_func
which I don't really understand.我真的不明白。 By the way, I also tried a variation where the name of _inner_func was func as well, that didn't change things.
顺便说一句,我还尝试了一个变体,其中 _inner_func 的名称也是 func ,但这并没有改变。
As detailled in the comments, the pickle module has issues when dealing with decorators.正如评论中详述的那样,pickle 模块在处理装饰器时存在问题。 See this question for more details:
有关更多详细信息,请参阅此问题:
Pickle and decorated classes (PicklingError: not the same object) Pickle 和装饰类(PicklingError:不是同一个对象)
Use methodtools.lru_cache
not to create a new cache function in __init__
使用
methodtools.lru_cache
不要在__init__
创建新的缓存函数
import pickle
from methodtools import lru_cache
class Test:
@lru_cache(maxsize=None)
def func(self, x):
# In reality this will be slow-running
return x
if __name__ == '__main__':
t = Test()
print(pickle.dumps(t))
It requires to install methodtools via pypi:它需要通过pypi安装methodtools:
pip install methodtools
If anybody is interested, this can be solved by using getstate and setstate like this:如果有人感兴趣,这可以通过使用 getstate 和 setstate 来解决,如下所示:
from functools import lru_cache
from copy import copy
class Test:
def __init__(self):
self.func = lru_cache(maxsize=None)(self._inner_func)
def _inner_func(self, x):
# In reality this will be slow-running
return x
def __getstate__(self):
result = copy(self.__dict__)
result["func"] = None
return result
def __setstate__(self, state):
self.__dict__ = state
self.func = lru_cache(maxsize=None)(self._inner_func)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.