Python中的巨大稀疏矩阵

Question

I need to iteratively construct a huge sparse matrix in numpy/scipy. The intitialization is done within a loop:

from scipy.sparse import dok_matrix, csr_matrix

def foo(*args):
    dim_x = 256*256*1024
    dim_y = 128*128*512
    matrix = dok_matrix((dim_x, dim_y))    

    for i in range(dim_x):
        # compute stuff in order to get j
        matrix[i, j] = 1.
    return matrix.tocsr()

Then i need to convert it to a csr_matrix, because of further computations like:

matrix = foo(...)
result = matrix.T.dot(x)

At the beginning this was working fine. But my matrices are getting bigger and bigger and my computer starts to crash. Is there a more elegant way in storing the matrix?

Basically i have the following requirements:

• The matrix needs to store float values form 0. to 1.
• I need to compute the transpose of the matrix
• I need to compute the dot product with a x_dimensional vector
• The matrix dimensions can be around 1*10^9 x 1*10^8

My ram-storage is exceeding. I was reading several posts on stack overflow and the rest of the internet ;) I found PyTables, which isn't really made for matrix computations... etc.. Is there a better way?

Answer 1

You may have hit the limits of what Python can do for you, or you may be able to do a little more. Try setting a datatype of np.float32 , if you're on a 64 bit machine, this reduced precision may reduce your memory consumption. np.float16 may help you on memory even further, but your calculations may slow down (I've seen examples where processing may take 10x the amount of time):

    matrix = dok_matrix((dim_x, dim_y), dtype=np.float32)    

or possibly much slower, but even less memory consumption:

    matrix = dok_matrix((dim_x, dim_y), dtype=np.float16)    

---

Another option: buy more system memory.

---

Finally, if you can avoid creating your matrix with dok_matrix , and can create it instead with csr_matrix (I don't know if this is possible for your calculations) you may save a little overhead on the dict that dok_matrix uses.

Answer 2

For your case I would recommend using the data type np.int8 (or np.uint8 ) which require only one byte per element:

matrix = dok_matrix((dim_x, dim_y), dtype=np.int8)

Directly constructing the csr_matrix will also allow you to go further with the maximum matrix size:

from scipy.sparse import csr_matrix

def foo(*args):
    dim_x = 256*256*1024
    dim_y = 128*128*512
    row = []
    col = []

    for i in range(dim_x):
        # compute stuff in order to get j
        row.append(i)
        col.append(j)
    data = np.ones_like(row, dtype=np.int8)

    return csr_matrix((data, (row, col)), shape=(dim_x, dim_y), dtype=np.int8)