Terminating try block in python after n seconds

Question

I am trying to impose a TimeoutException on a try statement after n seconds. I have found a library which handles this called signal which would be perfect but I'm running into an error I have a hard time getting around. ( This similar SO question is answered with the signal library.)

This is the boiled down code representing the problem:

import multiprocessing
from multiprocessing.dummy import Pool

def main():
    listOfLinks = []
    threadpool = Pool(2)
    info = threadpool.starmap(processRunSeveralTimesInParallel,zip(enumerate(listOfLinks)))
    threadpool.close()

def processRunSeveralTimesInParallel(listOfLinks):
    #The following is pseudo code representing what I would like to do:
    loongSequenceOfInstructions()
    for i in range(0,10):
        try for n seconds:
            doSomething(i)
        except (after n seconds):
            handleException()

    return something

When implementing the above question's solution with the signal library, I get the following error:

File "file.py", line 388, in main
    info = threadpool.starmap(processRunSeveralTimesInParallel,zip(enumerate(listOfLinks)))
  File "/Users/user/anaconda3/envs/proj/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/user/anaconda3/envs/proj/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/user/anaconda3/envs/proj/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/user/anaconda3/envs/proj/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "file.py", line 193, in processRunSeveralTimesInParallel
    signal.signal(signal.SIGALRM, signal_handler)
  File "/Users/user/anaconda3/envs/proj/lib/python3.8/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

Any idea how to cap the time just on a try block within a method run as a thread? Thank you!

Important information:

• I am using the multiprocessing library to run several processes at the same time in parallel. From the error statement above, I suspect that the signal and multiprocessing libraries conflict.
• The methods in the try statement are selenium (find_element_by_xpath) methods . However there are no timeout arguments available.

Answer 1

Newly Updated Answer

If you are a way of looking for timing out without using signals, here is one way. First, since you are using threading, let's make it explicit and let's use the concurrent.futures module, which has a lot of flexibility.

When a "job" is submitted to the pool executor, a Future instance is returned immediately without blocking until a result call is made on that instance. You can specify a timeout value such that if the result is not available within the timeout period, an exception will be thrown. The idea is to pass to the worker thread the ThreadPoolExecutor instance and for it to run the critical piece of code that must be completed within a certain time period within its own worker thread. A Future instance will be created for that timed code but this time the result call will specify a timeout value:

from concurrent.futures import ThreadPoolExecutor, TimeoutError
import time


def main():
    listOfLinks = ['a', 'b', 'c', 'd', 'e']
    futures = []
    """
    To prevent timeout errors due to lack of threads, you need at least one extra thread
    in addition to the ones being created here so that at least one time_critical thread
    can start. Of course, ideally you would like all the time_critical threads to be able to
    start without waiting. So, whereas the minimum number of max_workers would be 6 in this
    case, the ideal number would be 5 * 2 = 10.
    """
    with ThreadPoolExecutor(max_workers=10) as executor:
        # pass executor to our worker
        futures = [executor.submit(processRunSeveralTimesInParallel, tuple, executor) for tuple in enumerate(listOfLinks)]
        for future in futures:
            result = future.result()
            print('result is', result)


def processRunSeveralTimesInParallel(tuple, executor):
    link_number = tuple[0]
    link = tuple[1]
    # long running sequence of instructions up until this point and then
    # allow 2 seconds for this part:
    for i in range(10):
        future = executor.submit(time_critical, link, i)
        try:
            future.result(timeout=2) # time_critical does not return a result other than None
        except TimeoutError:
            handle_exception(link, i)
    return link * link_number


def time_critical(link, trial_number):
    if link == 'd' and trial_number == 7:
        time.sleep(3) # generate a TimeoutError


def handle_exception(link, trial_number):
    print(f'There was a timeout for link {link}, trial number {trial_number}.')


if __name__ == '__main__':
    main()

Prints:

result is
result is b
result is cc
There was a timeout for link d, trial number 7.
result is ddd
result is eeee

Using Threading and Multiprocessing

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, TimeoutError
import os
import time


def main():
    listOfLinks = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
    futures = []
    cpu_count = os.cpu_count()
    with ThreadPoolExecutor(max_workers=cpu_count) as thread_executor, ProcessPoolExecutor(max_workers=cpu_count) as process_executor:
        # pass executor to our worker
        futures = [thread_executor.submit(processRunSeveralTimesInParallel, tuple, process_executor) for tuple in enumerate(listOfLinks)]
        for future in futures:
            result = future.result()
            print('result is', result)


def processRunSeveralTimesInParallel(tuple, executor):
    link_number = tuple[0]
    link = tuple[1]
    # long running sequence of instructions up until this point and then
    # allow 2 seconds for this part:
    for i in range(10):
        future = executor.submit(time_critical, link, i)
        try:
            future.result(timeout=2) # time_critical does not return a result other than None
        except TimeoutError:
            handle_exception(link, i)
    return link * link_number


def time_critical(link, trial_number):
    if link == 'd' and trial_number == 7:
        time.sleep(3) # generate a TimeoutError


def handle_exception(link, trial_number):
    print(f'There was a timeout for link {link}, trial number {trial_number}.')


if __name__ == '__main__':
    main()

Prints:

result is
result is b
result is cc
There was a timeout for link d, trial number 7.
result is ddd
result is eeee
result is fffff
result is gggggg
result is hhhhhhh
result is iiiiiiii
result is jjjjjjjjj

Multiprocessing Exclusively

from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Process
import os
import time


def main():
    listOfLinks = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
    futures = []
    workers = os.cpu_count() // 2
    with ProcessPoolExecutor(max_workers=workers) as process_executor:
        # pass executor to our worker
        futures = [process_executor.submit(processRunSeveralTimesInParallel, tuple) for tuple in enumerate(listOfLinks)]
        for future in futures:
            result = future.result()
            print('result is', result)


def processRunSeveralTimesInParallel(tuple):
    link_number = tuple[0]
    link = tuple[1]
    # long running sequence of instructions up until this point and then
    # allow 2 seconds for this part:
    for i in range(10):
        p = Process(target=time_critical, args=(link, i))
        p.start()
        p.join(timeout=2) # don't block for more than 2 seconds
        if p.exitcode is None: # subprocess did not terminate
            p.terminate() # we will terminate it
            handle_exception(link, i)
    return link * link_number


def time_critical(link, trial_number):
    if link == 'd' and trial_number == 7:
        time.sleep(3) # generate a TimeoutError


def handle_exception(link, trial_number):
    print(f'There was a timeout for link {link}, trial number {trial_number}.')


if __name__ == '__main__':
    main()

Prints:

result is
result is b
result is cc
There was a timeout for link d, trial number 7.
result is ddd
result is eeee
result is fffff
result is gggggg
result is hhhhhhh
result is iiiiiiii
result is jjjjjjjjj