使用pickle进行模块化序列化（Python）

Question

I want to perform serialisation of some object graph in a modular way. That is I don't want to serialize the whole graph. The reason is that this graph is big. I can keep timestamped version of some part of the graph, and i can do some lazy access to postpone loading of the parts i don't need right now.

I thought i could manage this with metaprogramming in Python. But it seems that metaprogramming is not strong enough in Python.

Here's what i do for now. My graph is composed of several different objects. Some of them are instances of a special class. This class describes the root object to be pickled. This is where the modularity come in. Each time i pickle something it starts from one of those instances and i never pickle two of them at the same time. Whenever there is a reference to another instance, accessible by the root object, I replace this reference by a persistant_id, thus ensuring that i won't have two of them in the same pickling stream. The problem comes when unpickling the stream. I can found a persistant_id of an instance which is not loaded yet. When this is the case, i have to wait for the target instance to be loaded before allowing access to it. And i don't see anyway to do that :

1/ I tried to build an accessor which get methods return the target of the reference. Unfortunately, accessors must be placed in the class declaration, I can't assign them to the unpickled object. 2/ I could store somewhere the places where references have to be resolved. I don't think this is possible in Python : one can't keep reference to a place (a field, or a variable), it is only possible to keep a reference to a value.

My problem may not be clear. I'm still looking for a clear formulation. I tried other things like using explicit references which would be instances of some "Reference" class. It isn't very convenient though.

Do you have any idea how to implement modular serialisation with pickle ? Would i have to change internal behaviour of Unpickler to be able to remember places where i need to load the remaining of the object graph ? Is there another library more suitable to achieve similar results ?

Answer 1

Here's how I think I would go about this.

1. Have a module level dictionary mapping persistent_id to SpecialClass objects. Every time you initialise or unpickle a SpecialClass instance, make sure that it is added to the dictionary.

2. Override SpecialClass's __getattr__ and __setattr__ method, so that specialobj.foo = anotherspecialobj merely stores a persistent_id in a dictionary on specialobj (let's call it specialobj.specialrefs ). When you retrieve specialobj.foo , it finds the name in specialrefs, then finds the reference in the module-level dictionary.

3. Have a module level check_graph function which would go through the known SpecialClass instances and check that all of their specialrefs were available.

Answer 2

Metaprogramming is strong in Python; Python classes are extremely malleable. You can alter them after declaration all the way you want, though it's best done in a metaclass (decorator). More than that, instances are malleable, independently of their classes.

A 'reference to a place' is often simply a string. Eg a reference to object's field is its name. Assume you have multiple node references inside your node object. You could have something like {persistent_id: (object, field_name),..} as your unresolved references table, easy to look up. Similarly, in lists of nodes 'references to places' are indices.

BTW, could you use a key-value database for graph storage? You'd be able to pull nodes by IDs without waiting.