Limit function execution in Python

Question

There are a lot of similar questions and asnwers, but I still can't find reliable answer.

So, I have a function, that can possibly run too long. Function is private, in sense that I can not change it code.

I want to restrict its execution time to 60 seconds. I tried following approaches:

1. Python signals. Don't work on Windows and in multithreaded envorinment (mod_wsgi).
2. Threads. Nice way, but thread can not be stopped, so that it lives even after raising TimeoutException .
3. multiprocessing python module. I have problems with pickling and I don't know how to solve them. I want to make time_limit decorator and there are problems with importing required function in top-level. Function that is executed long is instance method, and wrapping it also doesn't help...

So, are there good solutions to the above problem? How to kill thread, that I started? How to use subprocesses and avoid problems with pickling? Is subprocess module of any help?

Thank you.

Answer 1

I think the multiprocessing approach is your only real option. You're correct that threads can't be killed (nicely) and signals have cross-platform issues. Here is one multiprocessing implementation:

import multiprocessing
import Queue

def timed_function(return_queue):
    do_other_stuff()
    return_queue.put(True)
    return

def main():

    return_queue = multiprocessing.Manager().Queue()

    proc = multiprocessing.Process(target=timed_function, args=(return_queue,))
    proc.start()

    try:

        # wait for 60 seconds for the function to return a value
        return_queue.get(timeout=60)

    except Queue.Empty:
        # timeout expired
        proc.terminate() # kill the subprocess
        # other cleanup

I know you said that you have pickling issues, but those can almost always be resolved with refactoring. For example, you said that your long function is an instance method. You can wrap those kinds of functions to use them with multiprocessing:

class TestClass(object):
    def timed_method(self, return_queue):
        do_other_stuff()
        return_queue.put(True)
        return

To use that method in a pool of workers, add this wrapper to the top-level of the module:

def _timed_method_wrapper(TestClass_object, return_queue):
    return TestClass_object(return_queue)

Now you can, for example, use apply_async on this class method from a different method of the same class:

def run_timed_method():
    return_queue = multiprocessing.Manager().Queue()
    pool = multiprocessing.Pool()
    result = pool.apply_async(_timed_method_wrapper, args=(self, return_queue))

I'm pretty sure that these wrappers are only necessary if you're using a multiprocessing.Pool instead of launching the subprocess with a multiprocessing.Process object. Also, I bet a lot of people would frown on this construct because you're breaking the nice, clean abstraction that classes provide, and also creating a dependency between the class and this other random wrapper function hanging around. You'll have to be the one to decide if making your code more ugly is worth it or not.

Answer 2

An answer to Is it possible to kill a process on Windows from within Python? may help: You need to kill that subprocess or thread: "Terminating a subprocess on windows"

Maybe also TerminateThread helps