How to unpickle an object whose class exists in a different namespace (python)?

Question

If I have a script that defines a class:

script = """

class myClass:
    def __init__(self):
        self.name = 'apple'
        self.color = 'green'

"""

and then exec this script in its own namespace dict:

NS = {}
exec script in NS

and then create an instance of the class and pickle it:

a = NS['myClass']()

import pickle

save = pickle.dumps(a)

Now if I try to unpickle it:

load = pickle.loads(save)

I get the error

AttributeError: 'module' object has no attribute 'myClass'

I gather that this doesn't work because python doesn't know where to find myClass in order to rebuild the object. But myClass does exist in the NS dict. Is there a way to tell pickle where to find the class for the object it is loading?

Answer 1

You can actually go one step further, and have the object reconstruct itself into whatever type you want.

import pickle
import copy_reg

class myClass(object):
    def __init__(self):
        self.apple = 'banana'

class otherclass(object):
    def __init__(self):
        self.apple = 'existential woe'

def pickle_an_object(o):
    print "pickling %s" % str(o)
    return otherclass, (o.apple,)

copy_reg.pickle(myClass, pickle_an_object)

foo = myClass()

s = pickle.dumps(foo)

del myClass
del otherclass

class otherclass(object):
    def __init__(self, appletype):
        self.apple = 'not %s' % appletype

o2 = pickle.loads(s)

print o2.apple

The basic idea is that you pack your class into a "trojan horse" of sorts, where its reconstruction causes an instantiation of a different class from what it originally was.

It does not matter what the otherclass on the pickling side contains. All that matters is that it exist at the same module path as the "destination" class - pickle is just putting a string representation of the module name into the serialized stream.

So, to break down what's happening in the above code in detail:

• We register a custom pickler for myClass . This can be done via copy_reg or the __reduce_ex__ function.
• Our custom pickler says "pickle this as an instance of otherclass " (which is a dummy. You do not need the "real" contents of otherclass on the pickling side, because all that goes into the pickle is the module/class name).
• We pickle the object and "send it across the wire", to where the real version of otherclass exists.
• On the remote side, otherclass is instantiated with the data from the tuple returned by the custom pickling function.

Python can be pretty powerful!

Answer 2

I discovered a solution this. It seems the problem is executing code in a dict prevents python from figuring out where the class is defined. The solution is to create an empty module, execute the code in the module, and then add the module to sys.modules so python knows about it.

script = """
class myClass:
    def __init__(self):
        self.name = 'apple'
        self.color = 'green'
"""

import imp, sys

moduleName = 'custom'

module = imp.new_module(moduleName)

exec script in module.__dict__

sys.modules[moduleName] = module

Now it is possible to pickle and unpickle an instance of the class:

import pickle
a = module.myClass()
s = pickle.dumps(a)
b = pickle.loads(s)