[英]Python: Pickle class object that has functions/callables as attributes
I have custom class which inherits from functools.partial我有自定义 class 继承自 functools.partial
from functools import partial
from typing import Callable
class CustomPartial(partial):
def __new__(cls, func_name: str, func: Callable, *args, **kwargs):
self=super(CustomPartial, cls).__new__(cls, func, *args, **kwargs)
self.func_name = func_name
return self
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
This code is fine as is for serial processing, ie I can create objects of this class as needed and call them as normal functions.这段代码对于串行处理来说很好,即我可以根据需要创建这个 class 的对象并将它们作为普通函数调用。
The issue I'm running into though is when I try and use one of these CustomPartial objects as the function input for joblib
Parallel processing.我遇到的问题是,当我尝试使用这些 CustomPartial 对象之一作为 function 输入以进行
joblib
并行处理时。 Based on the exceptions being throw基于抛出的异常
TypeError: CustomPartial.__new__() missing 1 required positional argument: 'func'
I've summarised that the issue is happening when trying to "un-serialize" between processes.我总结说,当尝试在进程之间“反序列化”时,问题正在发生。
The code below is a minimum working example of the issue.下面的代码是该问题的最小工作示例。 I've tried to serialize using
dill
and tried implementing the __setstate__
/ __getstate__
functions but nothing seems to be changing the exception being thrown.我尝试使用
dill
进行序列化并尝试实现__setstate__
/ __getstate__
函数,但似乎没有任何改变被抛出的异常。
import dill
from typing import Callable
from functools import partial
class CustomPartial(partial):
def __new__(cls, func_name: str, func: Callable, *args, **kwargs):
self=super(CustomPartial, cls).__new__(cls, func, *args, **kwargs)
self.func_name = func_name
return self
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
add = lambda x, y: x+y
add_ten = partial(add, y=10)
custom_partial = CustomPartial('add_ten', add_ten)
print(dill.loads(dill.dumps(add_ten)))
# functools.partial(<function <lambda> at 0x7f7647eefa30>, y=10)
try:
print(dill.loads(dill.dumps(custom_partial)))
except Error as err:
print(err)
# CustomPartial.__new__() missing 1 required positional argument: 'func'
Any help / direction towards resolving this issue would be greatly appreciated:)任何解决此问题的帮助/方向将不胜感激:)
Edit: the solution is complex because partial
uses __setstate__()
编辑:解决方案很复杂,因为
partial
使用__setstate__()
Didn't test it, but you probably need to override the method partial.__reduce__()
in your CustomPartial
class to match its __new__()
signature with an extra argument.没有测试它,但您可能需要覆盖
CustomPartial
class 中的方法partial.__reduce__()
以匹配其__new__()
签名和额外的参数。
This is the partial.__reduce__()
definition in Python 3.10:这是 Python 3.10 中的
partial.__reduce__()
定义:
def __reduce__(self):
return type(self), (self.func,), (self.func, self.args,
self.keywords or None, self.__dict__ or None)
You should include the extra argument/attribute in the second item of the returned tuple, which is passed as *args
to __new__()
when unpickling an object of this class.您应该在返回的元组的第二项中包含额外的参数/属性,当解开此 class 的 object 时,它作为
*args
传递给__new__()
。 Plus, as partial
uses __setstate__()
to set its __dict__
attribute, you'll need to take care of that, otherwise the func_name
attribute will be erased.另外,由于
partial
使用__setstate__()
来设置其__dict__
属性,您需要注意这一点,否则func_name
属性将被删除。 If you use at least Python 3.8, and if you want to preserve the original __setstate__()
method, you can use the sixth field of the reduce value to pass a callable that controls how the update is made.如果您至少使用 Python 3.8,并且如果您想保留原始的
__setstate__()
方法,则可以使用 reduce 值的第六个字段来传递控制如何进行更新的可调用对象。
Try to add this to your class:尝试将此添加到您的 class 中:
def __reduce__(self):
return (
type(self),
(self.func_name, self.func),
(self.func, self.args, self.keywords or None, self.__dict__ or None),
None,
None,
self._setstate
)
@staticmethod
def _setstate(obj, state):
func_name = obj.func_name
obj.__setstate__(state) # erases func_name
obj.func_name = func_name
Reference:https://docs.python.org/3/library/pickle.html#object.__reduce__参考:https://docs.python.org/3/library/pickle.html#object.__reduce__
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.