带有莳萝的模块中的 Pickle 类定义

Question

My module contains a class which should be pickleable, both instance and definition I have the following structure:

MyModule
|-Submodule
  |-MyClass

In other questions on SO I have already found that dill is able to pickle class definitions and surely enough it works by copying the definition of MyClass into a separate script and pickling it there, like this:

import dill as pickle

class MyClass(object):
    ...

instance = MyClass(...)
with open(..., 'wb') as file:
   pickle.dump(instance, file)

However, it does not work when importing the class:

Pickling:

from MyModule.Submodule import MyClass
import dill as pickle

instance = MyClass(...)
with open(.., 'wb') as file:
    pickle.dump(instance, file)

Loading:

import dill as pickle

with open(..., 'rb') as file:
    instance = pickle.load(file)

>>> ModuleNotFoundError: No module named 'MyModule'

I think the class definition is saved by reference, although it should not have as per default settings in dill. This is done correctly when MyClass is known as __main__.MyClass , which happens when the class is defined in the main script.

I am wondering, is there any way to detach MyClass from MyModule ? Any way to make it act like a top level import ( __main__.MyClass ) so dill knows how to load it on my other machine?

Relevant question: Why dill dumps external classes by reference no matter what

Answer 1

I'm the dill author. This is a duplicate of the question you refer to above. The relevant GitHub feature request is: https://github.com/uqfoundation/dill/issues/128 .

I think the larger issue is that you want to pickle an object defined in another file that is not installed. This is currently not possible, I believe.

As a workaround, I believe you may be able to pickle with dill.source by extracting the source code of the class (or module) and pickling that dynamically, or extracting the source code and compiling a new object in __main__ .

Answer 2

I managed to save the instance and definition of my class using the following dirty hack:

class MyClass(object):
    def save(path):
        import __main__

        with open(__file__) as f:
            code = compile(f.read(), "somefile.py", 'exec')
            globals = __main__.__dict__
            locals = {'instance': self, 'savepath': path}
            exec(code, globals, locals)

if __name__ == '__main__':
    # Script is loaded in top level, MyClass is now available under the qualname '__main__.MyClass'
    import dill as pickle

    # copy the attributes of the 'MyModule.Submodule.MyClass' instance to a bew 'MyClass' instance.
    new_instance = MyClass.__new__(MyClass)
    new_instance.__dict__ = locals()['instance'].__dict__

    with open(locals()['savepath'], 'wb') as f:       
        pickle.dump(new_instance, f)

Using the exec statement the file can be executed from within __main__ , so the class definition will be saved as well. This script should not be executed as main script without using the save function.

Answer 3

Dill indeed only stores definitions of objects in __main__ , and not those in modules, so one way around this problem is to redefine those objects in main:

def mainify(obj):
    import __main__
    import inspect
    import ast

    s = inspect.getsource(obj)
    m = ast.parse(s)
    co = compile(m, '<string>', 'exec')
    exec(co, __main__.__dict__)

And then:

from MyModule.Submodule import MyClass
import dill as pickle

mainify(MyClass)
instance = MyClass(...)
with open(.., 'wb') as file:
    pickle.dump(instance, file)

Now you should be able to load the pickle from anywhere, even where the MyModule.Submodule is not available.