Async function call in python

Question

In python I want to create an async method in a class that create a thread without blocking the main thread. When the new thread finish, I return a value from that function/thread.

For example the class is used for retrieve some information from web pages. I want run parallel processing in a function that download the page and return a object.

class WebDown:
    def display(self, url):
        print 'display(): ' + content

    def download(self, url):
        thread = Thread(target=self.get_info)
        # thread join
        print 'download(): ' + content
        # return the info

    def get_info(self, url):
        # download page
        # retrieve info
        return info

if __name__ == '__main__':
    wd = WebDown()
    ret = wd.download('http://...')
    wd.display('http://...')

I this example, in order I call download() for retrieve the info, after display() for print others information. The print output should be

display(): foo, bar, ....
download(): blue, red, ....

Answer 1

One way to write asynchronous, non blocking code in python involves using Python's Twisted. Twisted does not rely on multithreading but uses multiprocessing instead . It gives you convenient way to create Deferred objects, adding callbacks and errbacks to them. The example you give would look like this in Twisted, I'm using treq (Twisted Requests) library which makes generating requests a little quicker and easier:

from treq import get
from twisted.internet import reactor

class WebAsync(object):
    def download(self, url):
        request = get(url)
        request.addCallback(self.deliver_body)

    def deliver_body(self, response):
        deferred = response.text()
        deferred.addCallback(self.display)
        return deferred

    def display(self, response_body):
        print response_body
        reactor.stop()

if __name__ == "__main__":
    web_client = WebAsync()
    web_client.download("http://httpbin.org/html")
    reactor.run()

Both 'download' and 'deliver_body' methods return deferreds , you add callbacks to them that are going to be executed when results is available.

Answer 2

I would simply use request and gevent called grequests .

import grequests
>>> urls = [
    'http://...',
    'http://...'
]
>>> rs = (grequests.get(u) for u in urls)
>>> grequests.map(rs)
[<Response [200]>, <Response [200]>]