python on Ubuntu: how to catch out of memory exceptions?

Question

I have the following block in my python script that is running on an Ubuntu AWS EC2 instance:

try:
    data = json.loads(line)
    # further processing of data
except Exception, e:  
    # something went bad

line is a string ingested from a text file. In most cases it gets processed fine. From time to time, I get lines that are humongous. In this case the script dies ("-9") and dmesg -T shows a message like [Tue Jan 8 16:10:48 2013] Out of memory: Kill process 13609 (python) score 910 or sacrifice child

What I don't understand is why instead of crashing it does not catch an exception in the try-except block. And is it possible to make changes in this block so that the script does not crash but raises and exception? Thx

Answer 1

Your Python process is being killed by the Kernel OOM (out-of-memory) killer ( docs ). This is killing Python with SIGKILL , so Python does not have any chance to respond to this event...hence it is not able to generate any exceptions for your code.

You may need to run your process in an environment with more resources (so that you're not using up such a large percentage of system memory), or you may need to place limits on the size of the JSON data you can read.

One of the links I posted below mentions ijson , which is an iterative JSON parser. This might be more memory efficient than the standard JSON parsers, but I haven't tried it.