Non-blocking/async URL fetching in Tornado request

Question

I apologize if this has been answered elsewhere; I've looked around and haven't found a definitive answer.

I'd like to use Tornado to accept an HTTP request with querystring parameters, use those params to call a NOAA web service to fetch weather data, process/parse the NOAA response, then return the final data to the user.

I'm looking at Tornado because I can't count on the latency or availability of the web service request and I need the calls to be non-blocking. (otherwise I'd just use Django)

I also want to make sure I can set an appropriate timeout on the NOAA request so I can give up as necessary.

Note: I'm also open to using Twisted, though it seems to have a much steeper learning curve and my needs feel pretty simple. (I would do this in Node.js, but I'm much more comfortable handling the parsing requirements in Python)

Thanks in advance for anyone who can help point me in the right direction.

I will open-source the server process once finished and credit anyone who contributes examples or RTFM links to the appropriate documentation.

Answer 1

I've extracted code sample from my project. It's not perfect, but it gives an idea how to use Tornadp's AsyncHTTPClient

@tornado.gen.engine
def async_request(self, callback, server_url, method=u'GET', body=None, **kwargs):
    """
    Make async request to server
    :param callback: callback to pass results
    :param server_url: path to required API
    :param method: HTTP method to use, default - GET
    :param body: HTTP request body for POST request, default - None
    :return: None
    """
    args = {}

    if kwargs:
        args.update(kwargs)

    url = '%s?%s' % (server_url, urlencode(args))
    request = tornado.httpclient.HTTPRequest(url, method, body=body)

    http = tornado.httpclient.AsyncHTTPClient()
    response = yield tornado.gen.Task(http.fetch, request)

    if response.error:
        logging.warning("Error response %s fetching %s", response.error, response.request.url)
        callback(None)
        return
    data = tornado.escape.json_decode(response.body) if response else None
    callback(data)