Why is the execution of a memory-intensive recursive function suppressed without an error?

Question

I have a problem with a function in a file that I am executing. It is expected to produce an IndexError , which it usually does.

However, when N defined as below becomes large enough the error is not shown and seems to be suppressed. I would like to know why this is happening. Why isn't some error thrown?

from collections import Counter
from sys import setrecursionlimit

def correct(events: list, lens: list=[]):

    counts = Counter(events).values()
    present = []
    while events[0] not in present:
        present.append(events.pop(0))
    lens.append(len(present))
    correct(events, lens)

if __name__ == '__main__':

    N = 50000  # enough in my case, increase if the problem does not occur
    setrecursionlimit(10000)
    result = correct([711321] * N)

I am executing the file on Windows 10 via PowerShell. After the execution of the file, no output is produced. Nonetheless, its return status $? is False , indicating that there is something wrong with the execution.

Note that running this piece of code in a Python shell closes it forcefully.

Answer 1

This is because the sys.setrecursionlimit sets the recursion limit, which the maximum depth on the interpreter stack, to at most the system's recursion limit, which in turn you can get by sys.getrecursionlimit. Actually the CPython interpreter will set the limit to be somewhat bigger than you give it, because when you overflow over the recursive limit, some cleanup function will come in and they will need some stack to recurse too.

When you put it far too high, and because the highest possible limit is platform-dependent, The interpreter will overflow while cleaning up an overflow, and then the interpreter will exit with a fatal error (the interpreter will exit with error, and not your code) "Py_FatalError("Cannot recover from stack overflow.");". And because the interpreter exits, it will not print an error message to you.

location in in CPython

if (tstate->overflowed) {
        if (tstate->recursion_depth > recursion_limit + 50) {
            /* Overflowing while handling an overflow. Give up. */
            Py_FatalError("Cannot recover from stack overflow.");
        }
return 0;

Answer 2

This is the code you are looking for:

def correct(events: list, lens: list=[]):
    while True:
        present = []
        while events[0] not in present:
            present.append(events.pop(0))
        lens.append(len(present))



if __name__ == '__main__':

    N = 50000  # enough in my case, increase if the problem does not occur
    result = correct([711321] * N)

This code just loops until the IndexError terminates the program.

The reason your original code returned nothing is that you set the recursion limit too high and python itself ran out of stack space and crashed.

The first 'fix' was to not mess with the recursion limit. The second fix was to not make recursive calls, but instead use a while loop.