Limit connections for particular view in Tornado

Question

I have view that takes a lot of memory and is asynchronous. Can I limit count of connections simultaneously working inside handler function (like critical section with N max workers inside).

Is this possible in Tornado?

Like:

@tornado.web.asynchronous
def get(self):
    with critical_section(count=5):
    # some code

Thanks

Answer 1

Toro provides synchronization primitives similar to those found in the threading module for Tornado coroutines. You could use its BoundedSemaphore to gate entry to the handler body:

# global semaphore
sem = toro.BoundedSemaphore(5)

@gen.coroutine
def get(self):
    with (yield sem.acquire()):
        # do work

Answer 2

Short answer:

As far as I understand Tornado and other frameworks that use Future/Deferred/generator based concurrency, this is not possible. However, it should definitely be possible using higher-order functions, ie a critical_section() helper function that takes the body of the with -block as a parameter.

Long answer:

To my best knowledge, Tornado's concurrency works very much like that of Twisted; which means non-blocking calls are limited to using Future s and yield (based on Twisted's @inlineCallbacks or whatever is the equivalent in Tornado).

In order to implement a critical_section context manager, it would have to cooperate with the reactor internally; this can only happen using callbacks or yield . However, neither is composable with context managers.

I'd actually already thrown up some code until I remembered this. This is the code I came up with:

import sys
from contextlib import contextmanager
from collections import defaultdict

from tornado.concurrent import Future


_critical_sections = defaultdict(lambda: (0, []))


@contextmanager
def critical_section(count):
    # get the code location of the critical section
    frame = sys._getframe()
    orig_caller = frame.f_back.f_back
    lineno = orig_caller.f_lineno
    filename = orig_caller.f_code.co_filename
    loc = (filename, lineno)

    count, waiters = _critical_sections[loc]
    if count > 5:
        future = Future()
        _critical_sections[loc] = (count + 1, waiters + [future])
        # XXX: not possible; either have to set a callback or use yield, but
        # then this context manager itself would not work as you'd expect:
        future.wait()  # <---- not possible in Tornado nor Twisted; only in Gevent/Eventlet
        fn(*args, **kwargs)
    else:
        _critical_sections[loc] = (count + 1, waiters)
        try:
            yield
        finally:
            count, waiters = _critical_sections[loc]
            _, w_future = waiters[0]
            _critical_sections[loc] = (count, waiters[1:])
            w_future.set_result(None)

(I have not tested it in anyway, and it's not runnable on Tornado anyway.)

Now, if you're happy with the proposed approach, here's something to get you started (or maybe it even works out of the box):

...

def _critical_section(count, fn, *args, **kwargs):
    ...
    if count > 5:
        future = Future()
        future.add_done_callback(lambda _: fn(*args, **kwargs))
        _critical_sections[loc] = (count + 1, waiters + [future])
        # XXX: not possible; either have to set a callback or use yield, but
        # then this context manager itself would not work as you'd expect:
        return future
    else:
        _critical_sections[loc] = (count + 1, waiters)
        try:
            return fn()
        finally:
            ... # same

then you could just turn it into a decorator:

from functools import wraps

def critical_section(count):
    def decorate(fn):
        @wraps(fn)
        def ret(*args, **kwargs):
            return _critical_section(count, fn, *args, **kwargs)
        return ret
    return decorate

Usage:

@tornado.web.asynchronous
def get(self):
    @critical_section(count=5)
    def some_code():
        pass  # do stuff

Also, the code is using sys._getframe() , which has (at least) 2 implications:

• it will make the code slower when run on PyPy (until/unless PyPy has become able to JIT-compile functions that use sys._getframe ), but most of the time, it's an acceptable tradeoff when it comes to web code
• I don't think the code will work if you compile it to .pyc and remove the .py —it shouldn't then be able to determine the filename and line number of the calling code, so it will not (probably) be possible to uniquely distinguish the location of the critical section, in which case you'd have to use lock objects.

NOTE: The context manager version would be perfectly feasible on Gevent ( http://gevent.org ) or Eventlet.