Python中的缓冲/批处理序列化？

Question

I have an algorithm that iteratively creates a very large, highly nested dictionary. I would like to buffer parts of this dictionary and then periodically stream the buffer to disk so that I can re-create the whole dictionary at another time.

It seems like pickle is intended for one-pass serialization. Is there a way to serialize a dictionary in batches to a single output stream?

Answer 1

Ok, it looks like the following will partially solve the problem:

with open('file','ab') as f:
  while <stopping condition>:
    <generate (key,value) pair 'k'>
    pickle.dump(k,f)

Now, to reconstruct the whole dictionary, you just do the following:

with open('file','rb') as f:
    fullMapping = {}
        hasNext = True
        while hasNext:
            try:
                fullMapping.update(pickle.load(f))
            except:
                f.close()
                hasNext = False

This will reconstitute the full dictionary when run.