在Python中保存和加载大型字典的最快方法

Question

I have a relatively large dictionary. How do I know the size? well when I save it using cPickle the size of the file will grow approx. 400Mb. cPickle is supposed to be much faster than pickle but loading and saving this file just takes a lot of time. I have a Dual Core laptop 2.6 Ghz with 4GB RAM on a Linux machine. Does anyone have any suggestions for a faster saving and loading of dictionaries in python? thanks

Answer 1

Use the protocol=2 option of cPickle. The default protocol (0) is much slower, and produces much larger files on disk.

If you just want to work with a larger dictionary than memory can hold, the shelve module is a good quick-and-dirty solution. It acts like an in-memory dict, but stores itself on disk rather than in memory. shelve is based on cPickle, so be sure to set your protocol to anything other than 0.

The advantages of a database like sqlite over cPickle will depend on your use case. How often will you write data? How many times do you expect to read each datum that you write? Will you ever want to perform a search of the data you write, or load it one piece at a time?

If you're doing write-once, read-many, and loading one piece at a time, by all means use a database. If you're doing write once, read once, cPickle (with any protocol other than the default protocol=0) will be hard to beat. If you just want a large, persistent dict, use shelve.

Answer 2

Sqlite

It might be worthwhile to store the data in a Sqlite database. Although there will be some development overhead when refactoring your program to work with Sqlite, it also becomes much easier and performant to query the database.

You also get transactions, atomicity, serialization, compression, etc. for free.

Depending on what version of Python you're using, you might already have sqlite built-in.

Answer 3

I know it's an old question but just as an update for those who still looking for an answer to this question: The protocol argument has been updated in python 3 and now there are even faster and more efficient options (ie protocol=3 and protocol=4 ) which might not work under python 2. You can read about it more in the reference .

In order to always use the best protocol supported by the python version you're using, you can simply use pickle.HIGHEST_PROTOCOL . The following example is taken from the reference :

import pickle
# ...
with open('data.pickle', 'wb') as f:
    # Pickle the 'data' dictionary using the highest protocol available.
    pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

Answer 4

您可以测试压缩您的字典（有一些限制请参阅： 这篇文章 ）如果磁盘访问是瓶颈，它将是高效的。

Answer 5

That is a lot of data... What kind of contents has your dictionary? If it is only primitive or fixed datatypes, maybe a real database or a custom file-format is the better option?