python: fast lightweight persistence

Question

I'm writing a python program that runs on a rather slow phone (as well as other platforms). Reading in data on the phone with pickle takes about 2 seconds and simplejson is significantly slower. The phone runs python 2.5.4

The data are in various lists and dictionaries in a group of classes. I can convert to and from a json format relatively quickly. The bulk of the time seems to be in pickle or json processing, as just reading or writing the json or pickle file is much faster than using pickle or json.

The program has to read the entire dataset and to write it periodically.

I suppose I could create some way to translate the data to text and see if that's faster, but reinventing the wheel is seldom a worthwhile endeavor.

What other storage alternatives might make sense?

Answer 1

What about the marshal module? According to these benchmarks it is significantly faster than pickle. However , make very sure to read and understand the warnings at the top of the documentation.

Also, just to make sure: did you use cPickle ? If not, try that — it's significantly faster than pickle .

Alternatively, since your data are fairly simple, you could write your own using pyrex . If you want to go that route, the MsgPack serializer would be a good starting point.