Python - 高性能序列化数据的最佳方式？

Question

I need to serialize a data which requires high performance. Separated thread will be accessing it each second and must to load data to memory. There will be about 1000 - 10000 dictionary-like entries about user sessions (id, sessid, login date). Some data will be frequently updated because login time has some time.

These data will be shared between python server and Django application. I think to use pickle or its faster version cPickle. I also found marshal .

What is the best way to do that? Is cPickle efficient enough? Or maybe marshal?

EDIT : Very important thing is access time. It's going to be a realtime websocket server so I need very low delays. Is it faster to access cPickle data once a second or to connect to database like Redis?

Answer 1

A better approach may be to use some in-memory cache - memcached, if your needs are simple - or something with a bit more of a feature set, like redis.

Answer 2

redis still requires serializing any complex python object , so redis doesn't solve this problem unless you represent all your data as simple keys and simple values. redis is not a deserialization solution, it's just a data store for strings. and in any case, redis is one of the slower options: https://charlesleifer.com/blog/completely-un-scientific-benchmarks-of-some-embedded-databases-with-python/

Answer 3

Use a real database in memory. Don't use pickle, cPickle or marshal, or anything like that.