Python Numpy.ndarray.shape limit

Question

I want to create a matrix with Numpy in Python with the following code:

import numpy

result=numpy.zeros((20,20,20,30,30,30))

numpy.save('result',result)

I get the following error:

Traceback (most recent call last):  
File "numpy_memoryerror.py",
line 5, in <module>
    result=numpy.zeros((20,20,20,30,30,30))
MemoryError

If I use smaller dimensions like these:

result=numpy.ones((10,10,10,20,20,20))

then the code works.

Can somebody tell me the limit of the shape tuple?

Answer 1

Its not a fundamental limit of the shape tuple, its that you don't have enough memory (RAM) on your system, hence the MemoryError .

Again 20*20*20*30*30*30 is 216 million 64-bit (8 byte) floats or a little more than 1.6 GB of RAM. So do you have 1.6 GB of RAM free while running the script at that point? (Don't forget to all the RAM used by python, the OS, other running programs, etc.). If you are in linux/unix you can see how much free memory by typing free -m from the command prompt. In windows you can see free memory by going to the task manager. Furthermore, some OSes limit the amount of memory that a single process (like python) can allocate; eg, 32-bit windows only gives 2 GB of address space per process).

Compare that to 20*20*20*10*10*10, which is only ~0.06 GB (or just 27 times less memory).

If you don't need 8-byte floats, you could do

numpy.zeros(20,20,20,30,30,30, dtype='float32')

which would halve your memory footprint by using single-precision (32-bit) floats. By default numpy uses dtype='float64'.

Roughly speaking a 32-bit float has 8 digits of accuracy, a 64-bit float has 16 digits of accuracy. Meaning that 1+1e-8 appears as just 1 for 32-bit floats, and 1+1e-16 appears as 1 for 64-bit floats, but 1+1e-15 appears as 1.000000000000001 with 64-bit floats, but not 32-bit floats.