python asyncio cancel run_in_executor blocks mainthread

Question

I am seeing a unwanted behavior when launching a sync function via asyncio run_in_executor and then cancelling it. ThreadPoolExecutor (and also ProcessPoolExecutor) context manager will call the executor.shutdown on exit and wait for all pending work to be completed (with a join() on its threads).

I had expected the join to be managed asynchronously so other tasks might not be blocked, but what I've found is that the join is called synchronously.

Should the implementation of run_in_executor contemplate this? or is it something one should handle (like calling executor.shutdown on a thread)?

This example code starts 2 tasks and cancel the one running on the subthread, from the logs one can see that the subthread is the only code being executed after the cancel:

from time import sleep
from asyncio import (
    get_running_loop, 
    sleep as async_sleep,
    get_event_loop,
    create_task
)
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
​
​
def sync_watchdog():
    for _ in range(30):
        print(f'{datetime.now()} I am in a thread')
        sleep(1)
​
​
async def inthread():
    with ThreadPoolExecutor() as executor:
        await get_running_loop().run_in_executor(executor, sync_watchdog)
​
async def watchdog():
    for _ in range(30):
        print(f'{datetime.now()} I am an async watchdog')
        await async_sleep(1)
​
​
async def run():
    intread_task = create_task(inthread())
    watchdog_task = create_task(watchdog())
​
    await async_sleep(2)
​
    print('canceling task with subthread')
    intread_task.cancel()
​
    await async_sleep(10)
​
​
get_event_loop().run_until_complete(run())

That will output:

2021-10-27 17:08:16.796171 I am in a thread
2021-10-27 17:08:16.796248 I am an async watchdog
2021-10-27 17:08:17.797205 I am an async watchdog
2021-10-27 17:08:17.797310 I am in a thread
canceling task with subthread
2021-10-27 17:08:18.797990 I am an async watchdog
2021-10-27 17:08:18.798430 I am in a thread
2021-10-27 17:08:19.799037 I am in a thread
2021-10-27 17:08:20.800327 I am in a thread
2021-10-27 17:08:21.801521 I am in a thread
2021-10-27 17:08:22.802819 I am in a thread
2021-10-27 17:08:23.804014 I am in a thread
2021-10-27 17:08:24.805434 I am in a thread
2021-10-27 17:08:25.806640 I am in a thread
2021-10-27 17:08:26.807797 I am in a thread
2021-10-27 17:08:27.808880 I am in a thread
2021-10-27 17:08:28.810305 I am in a thread
2021-10-27 17:08:29.811247 I am in a thread
2021-10-27 17:08:30.811901 I am in a thread
2021-10-27 17:08:31.813083 I am in a thread
2021-10-27 17:08:32.814301 I am in a thread
2021-10-27 17:08:33.815499 I am in a thread
2021-10-27 17:08:34.816565 I am in a thread
2021-10-27 17:08:35.817920 I am in a thread
2021-10-27 17:08:36.818478 I am in a thread
2021-10-27 17:08:37.819603 I am in a thread
2021-10-27 17:08:38.820844 I am in a thread
2021-10-27 17:08:39.822042 I am in a thread
2021-10-27 17:08:40.822491 I am in a thread
2021-10-27 17:08:41.823690 I am in a thread
2021-10-27 17:08:42.824871 I am in a thread
2021-10-27 17:08:43.826012 I am in a thread
2021-10-27 17:08:44.826516 I am in a thread
2021-10-27 17:08:45.827698 I am in a thread
2021-10-27 17:08:46.829813 I am an async watchdog

Answer 1

Long story short: .shutdown() cancels only pending executor tasks, not already running.

That's because threading.Thread has no .cancel() or .abort() method. If you want to stop a loop in the running thread, create a flag, check it in every iteration, raise the flag if you need stopping.