从许多（大）字典中构建一个（大）稀疏矩阵

Question

I have a series of dicts of the form:

dict_k:{(i,j):d_ij}

where i and j are integers corresponding to the indices in the sparse matrix to be built and d_ij a float number. Dicionnaries can contain up to O(1 Million) values

I have about 160 such dictionaries, each dictionary is about 16 MegaBytes. Dictionaries may or may not contain duplicate keys with duplicate values, that is you could find the (0,0):1.25 key/value couple in two different dictionaries for example.

I want to build a sparse matrix containing out of these dictionaries. The entries of the matrix would be given by all of the {(i,j):d_ij} couples in all of the dictionaries.

My naive approach is to build a huge dictionary out of all dictionaries like so:

bigDict={}
for i in range(160):
    with open(dictionnary_path, "rb") as fp:   
        bigDict.update(pickle.load(fp))

Then retrieve the columns/row indices and their corresponding coefficients to build a scipy coo format sparse matrix with the line ( coo_matrix((coefficients, (rows, columns)), [shape=(M, N)]) ).

But this makes my computer freeze when building the huge dict, do you have any smarter way of doing it? My end end goal is to use this sparse matrix to perform matrix vector multiplications.

an example dict:

 {(0, 1704510): 0.125,
 (0, 1704511): 0.089,
 (0, 1704512): 0.044,
 (0, 1704513): 0.021,
 (0, 1704514): 0.037,
 (0, 1704515): 0.032,
 (0, 1704516): 0.021,
 (0, 1704517): 0.013,
 (0, 502593): 0.089,
 (0, 502594): 0.125,
 (0, 502595): 0.089,
 (0, 502596): 0.044,
 (0, 502597): 0.032,
 (0, 502598): 0.037,
 (0, 502599): 0.032,
 (0, 502600): 0.021,
 (0, 129844): 0.044,
 (0, 129845): 0.089,
 (0, 129846): 0.125,
 (0, 129847): 0.089,
 (0, 129848): 0.021,
 (0, 129849): 0.032,
 (0, 129850): 0.037,
 (0, 129851): 0.032,
 (0, 28314): 0.021,
 (0, 28315): 0.044,
 (0, 28316): 0.089,
 (0, 28317): 0.125,
 (0, 28318): 0.013,
 (0, 28319): 0.021,
 (0, 28320): 0.032,
 (0, 28321): 0.037,
 (0, 4917): 1.0,
 (0, 4918): 0.354,
 (0, 4919): 0.089,
 (0, 4920): 0.032,
 (0, 4921): 0.125,
 (0, 4922): 0.089,
 (0, 4923): 0.044,
 (0, 4924): 0.021,
 (0, 615): 0.354,
 (0, 616): 1.0,
 (0, 617): 0.354,
 (0, 618): 0.089,
 (0, 619): 0.089,
 (0, 620): 0.125,
 (0, 621): 0.089,
 (0, 622): 0.044,
 (0, 45): 0.089,
 (0, 46): 0.354,
 (0, 47): 1.0,
 (0, 48): 0.354,
 (0, 49): 0.044,
 (0, 50): 0.089,
 (0, 51): 0.125,
...
 (13, 675): 0.354,
 (13, 680): 0.089,
 (13, 684): 0.089,
 (13, 685): 0.125,
 ...}

The matrix is of size 5Millionx5Million approximately

Answer 1

pickle seems to create different objects for equal ints/floats, so probably most of your memory usage comes from the keys and values, not from the dictionaries themselves. You could try to deduplicate them, for example:

dedup = {}.setdefault
def d(x):
    return dedup(x, x)

bigDict={}
for i in range(160):
    with open(dictionnary_path, "rb") as fp:   
        for (x, y), z in pickle.load(fp).items():
            bigDict[d(x), d(y)] = d(z)