jupyter notebook 中的 asyncio 运行任务，失败时重试

Question

I would like to run a script.py file from a jupyter notebook:

%run script.py 

My task is to run a map-reduce on partitions of data on a dask cluster, asynchronously. The workers sometimes (like 5% of cases) randomly crash, due to a connection error, that we did not figure out yet, what is the reason for it. So, for now, I wanted to write a small loop around my async loop call that handles retrying up to n_tries_max times upon failure. The code in script.py looks like this:

# small wrapper for async stuff
async def my_func(args):
    await ...

# try running the function with retries on failure
n_tries_max = 3
n_try = 1
while n_try<=n_tries_max:
    try:
        loop = asyncio.get_event_loop()
        task = loop.create_task(my_func(args))
        asyncio.wait_for(task,timeout=None)
    except Exception as e:
        print(e)
        n_try += 1
        continue
    break

This works within jupyter notebook, however, I get a warning at the beginning:

/mt/users/jc/scripts/run_parallel_comp.py:62: RuntimeWarning: coroutine 'wait_for' was never awaited
  asyncio.wait_for(task,timeout=None)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

After the warning, the script keeps running and finishes correctly though. Just wanted to ask for the cause of this warning?

Thanks!

Answer 1

In my opinion, you should differentiate the meaning of async/await in Javascript and Python.

• When you call an awaitable function, you create a new coroutine object in Python.

• asyncio.create_task returns coroutine object to be waited using await keyword.

You should probably call this coroutine object using await keyword.

# small wrapper for async stuff
async def my_func(args):
    await ...

# try running the function with retries on failure
n_tries_max = 3
n_try = 1
while n_try<=n_tries_max:
    try:
        loop = asyncio.get_event_loop()
        task = loop.create_task(my_func(args))
        await task
    except Exception as e:
        print(e)
        n_try += 1
        continue
    break