How to generalize a function call which may be async, tornado coroutine, or normal?

Question

I have an application which has a library in multiple configurations:

• Python2.7 native
• Python2.7 tornado
• Python3.5 asyncio

Currently, I have code that is nearly identical against all three, but there are minor differences in how each function call are invoked. This means I have a ton of code duplication, because I have stuff like the following in many places:

#Python2.7native.py
def main(client):
   client.foo(args)
   client.bar(args)

#Python2.7tornado.py
@gen.coroutine
def main(client):
    yield client.foo(args)
    yield client.bar(args)

#Python3.5asyncio.py
async def main(client):
    await client.foo(args)
    await client.bar(args)

where client is a language specific implementation, supporting native python, asyncio, and tornado respectively. The API method calls are identical.

I am hoping to be able to somehow generalize this into a single method I can include in a shared file, which appropriately calls the various methods

I've thought about defining the methods in a separate file and using getattr to invoke the test properly, but this seems really messy.

Is there a good way to do this?

Answer 1

You can't do all of this in one function - how is client.foo() supposed to know whether it's being called from a "normal" synchronous application, or whether its caller is going to use yield or await . However, as long as you're willing to have Tornado as a dependency, you can avoid duplicating all your code three times.

In one module, client_async.py , implement your function(s) using Tornado's @gen.coroutine :

@gen.coroutine
def foo(args):
    yield something()
    yield something_else()
    raise gen.Return(another_thing())

In another, client_sync.py , wrap each of the functions from client_async.py in IOLoop.run_sync() for a thread-local IOLoop like this:

import client_async
import threading
import tornado.ioloop

class _LocalIOLoop(threading.local):
    def __init__(self):
        self.value = tornado.ioloop.IOLoop()
local_ioloop = _LocalIOLoop()

def foo(args):
    return local_ioloop.value.run_sync(lambda: foo_async.my_func(args))

Now you can use this code from all three environments. From normal synchronous code:

import client_sync

def main():
    x = client_sync.foo(args)

From Tornado @gen.coroutine :

import client_async

@gen.coroutine
def main():
    x = yield client_async.foo(args)

From async def and asyncio (note that the two are not synonymous - it is possible to use async def with Tornado without asyncio ):

# one-time initialization for Tornado/asyncio integration
import tornado.platform.asyncio
tornado.platform.asyncio.AsyncIOMainLoop().install()

import client_async

async def main():
    x = await client_async.foo(args)

Answer 2

Use @gen.coroutine and yield : This will work in all Python versions. A function decorated with gen.coroutine is a little slower than a native coroutine, but can be used in all the same scenarios.

For the synchronous case, use run_sync :

result = IOLoop.current().run_sync(main)