Why does numpy crash instantly upon an array insertion?

Question

I've come across some weird behaviour in a project of mine. Specifically, when this code is run:

import numpy as np 

coefficientMatrix = np.zeros([12500, 43750])
coefficientMatrix[229, 798] = 1.0942131827

my Python process crashes:

What can be wrong here?

System specs (in case that's relevant here): Windows 7 x64, 8Gb of RAM, Python 2.7 32-bit, numpy 1.9.2.

Answer 1

The reason why you would get a MemoryError when you assign to an element in coefficientMatrix, rather than when you create the array using np.zeros , is that most OSs (including Windows 7) use lazy memory allocation .

When you instantiate the array using np.zeros , Windows only allocates virtual memory address space rather than physical RAM. However, when you actually try to write to that chunk of memory then the OS will need to find enough physical memory to hold the array. If it fails to do so, you will get a MemoryError .

Since your Python process is 32bit, it can address a maximum of 4GB of memory (and possibly even less) . A 12500x43750 array of float64 values will take up 4.375GB of memory, so you simply can't have an array that big using 32bit Python.