How to save/load a dict of sparse list efficiently in Python?

Question

I have a dict with like a million keys and each value is a 10000-dim sparse list, eg,

dic = {'a': [0] * 10000, 'b': [0] * 1234 + [1, 2] + [0] * 8764, ...}

Dumping the object via pickle is not quite efficient as it takes a ton of disk spaces. Is there a way/package that I could save/load this kind of object efficiently?

Answer 1

You could save a sparse matrix using Scipy library as illustrated here . This function allows for compression as well, however, you need to represent your data in one of its supported formats.

You may also benefit by using Pandas data frames to store a sparse matrix ( more info ).

Answer 2

只需将列表对象替换为scipy稀疏矩阵 （例如，使用csr矩阵）。