Fire, Forget, and Return Value in Python3.7

Question

I have the following scenario: I have a python server that upon receiving a request, needs to parse some information, return the result to the user as quickly as possible, and then clean up after itself. I tried to design it using the following logic:

Consumer: *==*   (wait for result)   *====(continue running)=====...
              \                     / return
Producer:      *======(prase)====*=*
                                  \
Cleanup:                           *==========*

I've been trying to use async tasks and coroutines to make this scenario work with no avail. Everything I tried ends up with either the producer waiting for the cleanup to finish before returning, or the return killing the cleanup. I could in theory have the consumer call the cleanup after it displays the result to the user, but I refuse to believe Python doesn't know how to "fire-and-forget" and return.

For example, this code:

import asyncio

async def Slowpoke():
    print("I see you shiver with antici...")
    await asyncio.sleep(3)
    print("...pation!")

async def main():
    task = asyncio.create_task(Slowpoke())
    return "Hi!"

if __name__ == "__main__":
    print(asyncio.run(main()))
    while True:
        pass

returns:

I see you shiver with antici...
Hi!

and never gets to ...pation .

What am I missing?

Answer 1

I managed to get it working using threading instead of asyncio:

import threading
import time

def Slowpoke():
    print("I see you shiver with antici...")
    time.sleep(3)
    print("...pation")

def Rocky():
    t = threading.Thread(name="thread", target=Slowpoke)
    t.setDaemon(True)
    t.start()
    time.sleep(1)
    return "HI!"

if __name__ == "__main__":
    print(Rocky())
    while True:
        time.sleep(1)

Answer 2

asyncio doesn't seem particularly suited for this problem. You probably want simple threads:

The reasoning for this is that your task was being killed when the parent finished. By throwing a daemon thread out there, your task will continue to run until it finishes, or until the program exits.

import threading
import time

def Slowpoke():
    try:
        print("I see you shiver with antici...")
        time.sleep(3)
        print("...pation!")
    except:
        print("Yup")
        raise Exception()

def main():
    task = threading.Thread(target=Slowpoke)
    task.daemon = True
    task.start()
    return "Hi!"

if __name__ == "__main__":
    print(main())
    while True:
        pass

Answer 3

asyncio.run ...

[...] creates a new event loop and closes it at the end. [...]

Your coro, wrapped in task does not get a chance to complete during the execution of main .
If you return the Task object and and print it, you'll see that it is in a cancelled state:

async def main():
    task = asyncio.create_task(Slowpoke())
    # return "Hi!"
    return task

if __name__ == "__main__":
    print(asyncio.run(main()))

# I see you shiver with antici...
# <Task cancelled coro=<Slowpoke() done, defined at [...]>>

When main ends after creating and scheduling the task (and printing 'Hi!'), the event loop is closed, which causes all running tasks in it to get cancelled.

You need to keep the event loop running until the task has completed, eg by await ing it in main :

async def main():
    task = asyncio.create_task(Slowpoke())
    await task
    return task

if __name__ == "__main__":
    print(asyncio.run(main()))

# I see you shiver with antici...
# ...pation!
# <Task finished coro=<Slowpoke() done, defined at [..]> result=None>

Answer 4

(I hope I did properly understood your question. The ASCII image and the text description do not correspond fully in my mind. "Hi!" is the result and the "Antici..pation" is the cleanup, right? I like that musical too, BTW)

One of possible asyncio based solutions is to return the result asap. A return terminates the task, that's why it is necessary to fire-and-forget the cleanup. It must by accompanied with shutdown code waiting for all cleanups to finish.

import asyncio

async def Slowpoke():
    print("I see you shiver with antici...")
    await asyncio.sleep(3)
    print("...pation!")

async def main():
    result = "Hi!"
    asyncio.create_task(Slowpoke())
    return result

async def start_stop():
    # you can create multiple tasks to serve multiple requests
    task = asyncio.create_task(main())
    print(await task)

    # after the last request wait for cleanups to finish
    this_task = asyncio.current_task()
    all_tasks = [ 
        task for task in asyncio.all_tasks()
        if task is not this_task]
    await asyncio.wait(all_tasks)

if __name__ == "__main__":
    asyncio.run(start_stop())

---

Another solution would be to use other method (not return) to deliver the result to the waiting task, so the cleanup can start right after parsing. A Future is considered low-level, but here is an example anyway.

import asyncio

async def main(fut):
    fut.set_result("Hi!")
    # result delivered, continue with cleanup
    print("I see you shiver with antici...")
    await asyncio.sleep(3)
    print("...pation!")

async def start_stop():
    fut = asyncio.get_event_loop().create_future()
    task = asyncio.create_task(main(fut))
    print(await fut)

    this_task = asyncio.current_task()
    all_tasks = [ 
        task for task in asyncio.all_tasks()
        if task is not this_task]
    await asyncio.wait(all_tasks)

if __name__ == "__main__":
    asyncio.run(start_stop())