How to catch exceptions in a python run_in_executor method call

Question

How can i raise the exception in the run_long_thing() function called with the run_in_executor? It looks like it is being swallowed. I don't need the result of the function in the blocking code. It is basically a fire and forget function, but still i need to catch the exceptions if there are any...

import asyncio
import time


def fire_and_forget(task, *args, **kwargs):
    loop = asyncio.get_event_loop()
    if callable(task):
        #if threadpoolworker is set to None,
        #the max_workers will default to the number of processors on the machine, multiplied by 5
        return loop.run_in_executor(None, task, *args, **kwargs)
    else:    
        raise TypeError('Task must be a callable.')


async def run_long_thing(sleep):
    print("Doing long thing... {:}".format(sleep))
    time.sleep(sleep)
    print("Done doing long thing. {:}".format(sleep))
    raise Exception("sh*t happens")


def do_it():
    print("Starting my main thing...")
    print("Calling my long thing...")
    for i in range(0,10,1):
        try:
            fire_and_forget(run_long_thing, i)
            print(i)
            print("Pom pi dom...")
            time.sleep(0.1)
            print("POOOOM Pom pi dom...")
        except:
            print("can i see the sh*t?")

do_it()

Answer 1

first of all, if you call time.sleep you'll never end up running the asyncio event loop so no results will get detected. instead of calling time.sleep in do_it you're better off doing something like

asyncio.get_event_loop().run_until_complete(asyncio.sleep(0.1))

Now, the return from run_in_executor is a future. If you don't mind writing an async def and using create_task on your asyncio loop you could do something like

async def run_long_thing(thing, *args):
    try: await asyncio.get_event_loop().run_in_executor(None, thing, *args)
    except:
        #do stuff

But more in line with your current code you can attach an exception callback

def callback(future):
if future.exception(): #your long thing had an exception
        # do something with future.exception()

then when you call run_in_executor:

future = asyncio.get_event_loop().run_in_executor(None, fun, *args)
future.add_done_callback(callback)

Then callback will be called whenever your executor task completes. future.result() will contain the result if it is no an exception, and future.exception() will give you back any raised exception