pickling lru_cached function on object

Question

As part of parallellizing some existing code (with multiprocessing), I run into the situation that something similar to the class below needs to be pickled.

Starting from:

import pickle
from functools import lru_cache

class Test:
    def __init__(self):
        self.func = lru_cache(maxsize=None)(self._inner_func)

    def _inner_func(self, x):
        # In reality this will be slow-running
        return x

calling

t = Test()
pickle.dumps(t)

returns

_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000190454A7AC8>: it's not the same object as __main__.Test._inner_func

which I don't really understand. By the way, I also tried a variation where the name of _inner_func was func as well, that didn't change things.

Answer 1

As detailled in the comments, the pickle module has issues when dealing with decorators. See this question for more details:

Pickle and decorated classes (PicklingError: not the same object)

Answer 2

Use methodtools.lru_cache not to create a new cache function in __init__

import pickle
from methodtools import lru_cache

class Test:
    @lru_cache(maxsize=None)
    def func(self, x):
        # In reality this will be slow-running
        return x

if __name__ == '__main__':
    t = Test()
    print(pickle.dumps(t))

It requires to install methodtools via pypi:

pip install methodtools

Answer 3

If anybody is interested, this can be solved by using getstate and setstate like this:

from functools import lru_cache
from copy import copy


class Test:
    def __init__(self):
        self.func = lru_cache(maxsize=None)(self._inner_func)

    def _inner_func(self, x):
        # In reality this will be slow-running
        return x

    def __getstate__(self):
        result = copy(self.__dict__)
        result["func"] = None
        return result

    def __setstate__(self, state):
        self.__dict__ = state
        self.func = lru_cache(maxsize=None)(self._inner_func)