python asyncio add_done_callback with async def

Question

I have 2 functions: The first one, def_a , is an asynchronous function and the second one is def_b which is a regular function and called with the result of def_a as a callback with the add_done_callback function.

My code looks like this:

import asyncio

def def_b(result):
    next_number = result.result()
    # some work on the next_number
    print(next_number + 1)

async def def_a(number):
    await some_async_work(number)
    return number + 1

loop = asyncio.get_event_loop()
task = asyncio.ensure_future(def_a(1))
task.add_done_callback(def_b)
response = loop.run_until_complete(task)
loop.close()

And it's work perfectly.

The problem began when also the second function, def_b , became asynchronous. Now it looks like this:

async def def_b(result):
    next_number = result.result()
    # some asynchronous work on the next_number
    print(next_number + 1)

But now I can not provide it to the add_done_callback function, because it's not a regular function.

My question is- Is it possible and how can I provide def_b to the add_done_callback function if def_b is asynchronous?

Answer 1

add_done_callback is considered a "low level" interface. When working with coroutines, you can chain them in many ways, for example:

import asyncio


async def my_callback(result):
    print("my_callback got:", result)
    return "My return value is ignored"


async def coro(number):
    await asyncio.sleep(number)
    return number + 1


async def add_success_callback(fut, callback):
    result = await fut
    await callback(result)
    return result


loop = asyncio.get_event_loop()
task = asyncio.ensure_future(coro(1))
task = add_success_callback(task, my_callback)
response = loop.run_until_complete(task)
print("response:", response)
loop.close()

Keep in mind add_done_callback will still call the callback if your future raises an exception (but calling result.result() will raise it).

Answer 2

This only works for one future job, if you have multiple async jobs, they will blocks each other, a better way is using asyncio.as_completed() to iterate future list:

import asyncio

async def __after_done_callback(future_result):
    # await for something...
    pass

async def __future_job(number):
    await some_async_work(number)
    return number + 1

loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(__future_job(x)) for x in range(100)]  # create 100 future jobs

for f in asyncio.as_completed(tasks):
    result = await f
    await __after_done_callback(result)

loop.close()

Answer 3

You can try the aiodag library. It's a very lightweight wrapper around asyncio that abstracts away some of the async plumbing that you usually have to think about. From this example you won't be able to tell that things are running asynchronously since it's just 1 task that depends on another, but it is all running async.

import asyncio
from aiodag import task

@task
async def def_b(result):
    # some asynchronous work on the next_number
    print(result + 1)

@task
async def def_a(number):
    await asyncio.sleep(number)
    return number + 1

async def main():
    a = def_a(1)
    b = def_b(a)  # this makes task b depend on task a
    return await b

loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
response = loop.run_until_complete(main())