Tornado request response time too long even with gen.coroutine

Question

I'm new to asynchronous programming and being messed with it.

I got confused when I decorated the RequestHandler with gen.coroutine but found the request was still blocked.

Here is a brief code, with python 2.7.11 and tornado 4.4.1

@gen.coroutine
def store_data(data):
    try:
       # parse_data
       ...
    except ParseError as e:
       logger.warning(e)
       return
    yield motor.insert_many(parsed_data)  # asynchronous mongo
    print motor.count()

class MainHandler(RequestHandler):
    @gen.coroutine
    def post(self):
        try:
            some_argument = int(self.get_argument("some", 0))
            data = self.request.body
        except Exception:
            self.write("Improper Argument")
            self.finish()
            return
        IOLoop.current().spawn_callback(lambda: store_data(data))
        self.write("Request Done")
        self.finish()

And I made a test with 10 threads. According to the response time in access log, I suppose some requests were blocked

[I 161222 15:40:22 web:1971] 200 POST /upload/ (::1) 9.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 8.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 8.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 7.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 8.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 9.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 8.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 9.00ms
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 701.00ms # Seem blocked
[I 161222 15:40:23 web:1971] 200 POST /upload/ (::1) 696.00ms # Seem blocked

Update

The traceback message of set_blocking_log_threshold(0.5)

File "********", line 74, in <dictcomp>
    data = [dict({"sid": sid}, **{key: value for key, value in i.iteritems()

The whole line of this code is

data = [dict({"sid": sid}, **{key: value for key, value in i.iteritems() if key in need_cols}) for i in v_data] 

And the unpacked logic is something like this

data = []
# `v_data` is a huge dict which could be considered as a mongo collection, and `i` as a mongo document
for i in v_data:  
    temp = {key: value for key, value in i.iteritems() if key in need_cols}  # discard some keys
    temp["sid"] = sid  # add same `sid` to all items
    data.append(temp)

I changed it to a generator

def data_generator(v_data, need_cols, sid):
    for i in v_data:  
        temp = {key: value for key, value in i.iteritems() if key in need_cols}  # discard some keys
        temp["sid"] = sid  # add same `sid` to all items
        yield temp

@gen.coroutine
def store_data(data):
    try:
       # parse_data
       ...
    except ParseError as e:
       logger.warning(e)
       return
    ge = data_generator(v_data, need_cols, sid)
    yield motor.insert_many(ge)  # asynchronous mongo
    print motor.count()

No threshold warning logs reported any more, but the response time seemed still blocked

[I 170109 17:26:32 web:1971] 200 POST /upload/ (::1) 3.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 2.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 4.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 3.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 3.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 2.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 354.00ms
[I 170109 17:26:33 web:1971] 200 POST /upload/ (::1) 443.00ms

Then I set the threshold to 0.2s. Got this message

  File "*******", line 76, in store_data
    increment = json.load(fr)
  File "/usr/local/python2.7/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/local/python2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/python2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/python2.7/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)

Now I've no idea how to make this statement asynchronous

Answer 1

I think the problem may be about how you're calling your store_data coroutine function. You have to call coroutines in a right way. As it is mention here :

In nearly all cases, any function that calls a coroutine must be a coroutine itself, and use the yield keyword in the call.

So store_data must be called like this yield store_data() not store_data() . Here

IOLoop.current().spawn_callback(lambda: store_data(data))

I guess you're using lambda because you want to give data to your function as an argument but you can do this with spawn_callback itself. You can try this:

IOLoop.current().spawn_callback(store_data, data)

Hope this will help.

Answer 2

Decorating a function with @gen.coroutine does not do any good if that function never yields .

Your post() method looks correct: it never blocks or does anything to interfere with the IOLoop . However, the IOLoop is a shared resource and anything that blocks it could cause the timing that you're seeing. I suspect that something you're not showing us in store_data (or elsewhere in the program) is blocking. To highlight where this blocking might be, call IOLoop.current().set_blocking_log_threshold(0.5) at the start of your program and it will log a stack trace when the IOLoop is blocked for half a second.