Python is unable to allocate more than about 500MB while running CV program (in Ubuntu on 8GB RAM Virtualbox on Mac)

Question

Before I more closely studied the following error message...

ERROR: 
Traceback (most recent call last):
  File "polarize.py", line 13, in <module>
    output = img.binarize(val).invert()
  File "/usr/local/lib/python2.7/dist-packages/SimpleCV/ImageClass.py", line 3045, in invert
return -self
  File "/usr/local/lib/python2.7/dist-packages/SimpleCV/ImageClass.py", line 4729, in __neg__
return Image(newbitmap, colorSpace=self._colorSpace)
  File "/usr/local/lib/python2.7/dist-packages/SimpleCV/ImageClass.py", line 1035, in __init__
    self._bitmap = cv.CreateImage(cv.GetSize(source), source.depth, 3)
error: Failed to allocate 614794416 bytes

I hadn't thought to convert the bytes figure there into megabytes. Well, turns out its something like 500MB.

I've now allocated well over 8GB of memory (16GB Macbook) to this VirtualBox and obviously that does not aide me anymore here.

I need more "room" for Python.

By the way, the file being loaded in is a 9MB PNG file which balloons into about 80MB when expanded as a png, and fully balloons to something like 800mb when it becomes an uncompressed TIFF bitmap. However, in this case, it looks like the full extent of the memory requirements (~500MB) are a bit less than that. I would like to allocate 2-3GB to Python if it needs it.

Answer 1

"How does my OS allocate memory to processes" is not really a programming question, but what the heck.

Ubuntu being a modern operating system, you don't have to give Python more memory. It will use the most room the OS will give it. Since you've allocated 8 GB to the VM, the OS has plenty of RAM to hand out.

I'm assuming you have 32-bit Python. On 32-bit Ubuntu you're limited to 3 GB. On 64-bit Ubuntu (still running 32-bit Python) you have nearly 4 GB. However, the hard limit for a 32-bit process is 4 GB. There just aren't any more addresses than that in 32 bits.

My expectation is that you are indeed hitting this 3 GB or 4 GB limit because something (your script or your library) is holding on to a reference to images after you no longer need them. It would only take three or four images remaining resident in memory to leave you unable to create any more. Without knowing more details about your actual program it's hard to suggest where the problem could lie.

A brute-force approach would be to convert to 64-bit Python running on 64-bit Ubuntu if you have 64-bit versions of any compiled libraries you are using.