Python taking too much of memory

Question

I am importing sparse matrices from .npz file. Below is the script of the code. The sparse matrices (Dx, Dy, ..., M) have 373248x373248 size with 746496 stored elements.

if runmode == 2:
        data = np.load('Operators2.npz', allow_pickle=True)

    Dx = data['Dx']
    Dy = data['Dy']
    Dz = data['Dz']
    Dxx = data['Dxx']
    Dyy = data['Dyy']
    Dzz = data['Dzz']
    Dxp = data['Dxp']
    Dyp = data['Dyp']
    Dzp = data['Dzp']
    M = data['M']
    del data

If I print one of the variable, for example Dx, I get below output:

array(<373248x373248 sparse matrix of type '<class 'numpy.float64'>'
    with 746496 stored elements in Compressed Sparse Column format>,
      dtype=object)

But my system memory goes up and the program crashes. The program crashes when I execute the below line of code. I do not get any error, but the program crashes.

DIV = Dx*u+Dy*v+Dz*w

Even if I execute below lines of code, the memory consumption goes up and the program crashes

DIV = data['Dx']*u+data['Dy']*v+data['Dz']*w

Here u,v,w has 373248x1 shape. The shape of DIV is 373248x1. Since Dx, Dy, Dz are sparse matrices, Dx*u does matrix-vector multiplication and gives a vector.

If in the same code, I actually compute Dx, Dy,...,M there is no problem with the memory. If I am computing Dx then the output is as below:

<373248x373248 sparse matrix of type '<class 'numpy.float64'>'
    with 746496 stored elements in Compressed Sparse Column format>

So I think the issue with creating an object while importing. Is there a way to avoid that? Or,am I doing something wrong while importing the sparse matrices? Thank you.

Answer 1

Make a sparse matrix:

In [38]: M = sparse.random(1000,1000,.2,'csr')                                                 

save it 3 different ways:

In [39]: from scipy import io                                                                  
In [40]: np.savez('Msparse.npz', M=M)                                                          
In [41]: sparse.save_npz('M1sparse',M)                                                         

In [43]: io.savemat('Msparse.mat', {'M':M})                                                    

file sizes:

In [47]: ll M1spa* Mspar*                                                                      
-rw-rw-r-- 1 paul 1773523 Feb  1 12:40 M1sparse.npz
-rw-rw-r-- 1 paul 2404208 Feb  1 12:41 Msparse.mat
-rw-rw-r-- 1 paul 2404801 Feb  1 12:39 Msparse.npz

Load the 3 matrices:

In [48]: M1=sparse.load_npz('M1sparse.npz')                                                    
In [49]: M2=np.load('Msparse.npz',allow_pickle=True)['M']                                      
In [50]: M3=io.loadmat('Msparse.mat')['M']                                                     
In [51]: M1                                                                                    
Out[51]: 
<1000x1000 sparse matrix of type '<class 'numpy.float64'>'
    with 200000 stored elements in Compressed Sparse Row format>
In [52]: M2                                                                                    
Out[52]: 
array(<1000x1000 sparse matrix of type '<class 'numpy.float64'>'
    with 200000 stored elements in Compressed Sparse Row format>,
      dtype=object)
In [53]: M3                                                                                    
Out[53]: 
<1000x1000 sparse matrix of type '<class 'numpy.float64'>'
    with 200000 stored elements in Compressed Sparse Column format>

M1 and M3 are the same - csr like M for save_npz , csc (the MATLAB format) for .mat .

M2 is has an object dtype wrapper.

In [54]: (M1*np.ones((1000,1))).shape                                                          
Out[54]: (1000, 1)
In [55]: (M3*np.ones((1000,1))).shape                                                          
Out[55]: (1000, 1)

This took a lot longer; and I almost don't dare look at the result.

In [56]: (M2*np.ones((1000,1))).shape                                                          
Out[56]: (1000, 1)

If I extract the matrix from the object array, the multiplication is fast

In [57]: (M2.item()*np.ones((1000,1))).shape                                                   
Out[57]: (1000, 1)
In [58]: (M2.item()*np.ones((1000,1))).dtype                                                   
Out[58]: dtype('float64')
In [59]: (M3*np.ones((1000,1))).dtype                                                          
Out[59]: dtype('float64')

Looking more closely at the M2 multiplication:

In [60]: (M2*np.ones((1000,1))).dtype                                                          
Out[60]: dtype('O')
In [61]: (M2*np.ones((1000,1)))[:2,:]                                                          
Out[61]: 
array([[<1000x1000 sparse matrix of type '<class 'numpy.float64'>'
    with 200000 stored elements in Compressed Sparse Row format>],
       [<1000x1000 sparse matrix of type '<class 'numpy.float64'>'
    with 200000 stored elements in Compressed Sparse Row format>]],
      dtype=object)

It's performing a M*1 multiplication for each element of ones - making 1000 sparse matrices. That's where your memory consumption is going.

In sum, when using savez it wraps each sparse matrix in an object dtype array, and does the pickle. So you shouldn't use `data['Dx'] directly

Dx = data['Dx']  # wrong
Dx = data['Dx'].item()    # right

Answer 2

I was able to solve this issue by using scipy.io.savemat() and scipy.io.loadmat() . I was using np.savz() and np.load() before and that was taking too much memory.

Answer 3

The calculation would be well-suited for in-place operations:

np.multiply(data['Dx'], u, out=data['Dx']
np.multiply(data['Dy'], v, out=data['Dy']
np.multiply(data['Dz'], w, out=data['Dz']
numpy.add(data['Dx'], data['Dy'], out=data['Dx'])
numpy.add(data['Dx'], data['Dz'], out=data['Dx'])

which creates no additional temporary arrays. If you also avoid loading the other variables that are not required for this specific computation, you will save additional memory. Doing the work inside a function is a good way to ensure you clean-up / free memory once the work is done. In your case it is possibly better to pay a read penalty and just read in the specific data you need for each such calculation. Though like hpaulj said, there's not much more magic you can find; it's a large dataset and will require lots of memory. Any way to reduce the size / resolution of the problem, or work it in smaller blocks?