尝试对Numpy Ndarray对象求和时，Python中出现内存错误

Question

I have a huge numpy ndarray (called mat and of the shape 700000 x 6000) of which I want to sum through the columns and find the nonzero indices.

I want to sum through it like so:

x =  np.sum(mat[:,y], axis=1)
indices = np.nonzero(x)

But the first line immediately gives me an instant Memory Error. Is there a way I can go around using np.sum and do it another way that makes this calculation possible?

Answer 1

You have two problems:

1. See Sven Marnach's comment, it is possible that your data set is too large for your hardware
2. See ajcr's comment, what you want to do is not feasible the way you try do do it because the notation mat[:,an_index] gives you back an array of dimensionality one, whose only axis is axis=0

Another problem is the nature of your array, if it is an array of floating point numbers the probability that the sum of 700,000 entries is exactly equal to zero is close to zero... it's not impossible of course, but unlikely for certain it is.

That said, if you can reduce your data set or improve your hardware, you can do like this

In [39]: a = np.zeros((10,5))

In [40]: for i in range(5): a[3,i]=1+2*i if i != 3 else 0.0

In [41]: a
Out[41]: 
array([[ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 1.,  3.,  5.,  0.,  9.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.]])

In [42]: np.sum(a,axis=0)
Out[42]: array([ 1.,  3.,  5.,  0.,  9.])

In [43]: np.nonzero(np.sum(a,axis=0))
Out[43]: (array([0, 1, 2, 4]),)

In [44]: