多进程返回由进程创建的对象

Question

我正在使用多处理库来加速 class 创建的过程。 我的代码的最小示例：

from matplotlib.backends.backend_qt5agg import FigureCanvas
class Custom_Class(FigureCanvas)
    .
    .
    .

def generate_class_func(list_of_dfs, arg1, arg2):
    list_of_custom_classes = list()
    for df in list_of_dfs:
        custom_class = Custom_Class(df, arg1, arg2)
        list_of_custom_classes.append(custom_class)
    return list_of_custom_classes

def main():
    import multiprocessing as mp
    with mp.Pool() as p:
        list_of_classes_list = p.starmap(generate_class_func, zip(list_of_dfs, repeat(arg1), repeat(arg2)))
        p.close()
        p.join()
    

if __name__ == '__main__':
    main()

但是，我得到了multiprocess.pool.MaybeEncodingError: Error sending result: ... Reason: 'TypeError("cannot pickle 'Custom_Class' object")'错误。 我也尝试了 pathos.ProcessingPool 模块，但错误是一样的。 如何使用多处理返回自定义对象？

Answer 1

我是dill和multiprocess的作者。 使事情更可序列化的一个简单技巧是使用multiprocess而不是multiprocessing 。 前者是后者的一个分支，它使用dill而不是pickle ，因此您可以立即获得序列化更多对象的能力，包括大多数自定义类。

>>> from matplotlib.pylab import FigureCanvasBase
>>> class Custom_Class(FigureCanvasBase):
...   def __init__(self, df, arg1, arg2):
...     super().__init__()
...     self.df = df
...     self.arg1 = arg1
...     self.arg2 = arg2
... 
>>> def generate_class_func(list_of_dfs, arg1, arg2):
...     list_of_custom_classes = list()
...     for df in list_of_dfs:
...         custom_class = Custom_Class(df, arg1, arg2)
...         list_of_custom_classes.append(custom_class)
...     return list_of_custom_classes
... 
>>> 
>>> import multiprocess as mp
>>> with mp.Pool() as p:
...   from itertools import repeat
...   list_of_classes_list = p.starmap(generate_class_func, zip([[0,1],[2,3],[4,5],[6,7]], repeat('arg1'), repeat('arg2')))
...   p.close()
...   p.join()
... 
>>> list_of_classes_list
[[<__main__.Custom_Class object at 0x11b35a390>, <__main__.Custom_Class object at 0x11b0fcb10>], [<__main__.Custom_Class object at 0x11b373110>, <__main__.Custom_Class object at 0x11ae1fa10>], [<__main__.Custom_Class object at 0x11b37b990>, <__main__.Custom_Class object at 0x11b3965d0>], [<__main__.Custom_Class object at 0x11b114f90>, <__main__.Custom_Class object at 0x11adfdb90>]]