Gevent：重试greenlet一次

Question

I'm doing a few dozen HTTP requests inside a Gevent pool.

The goal is to retry a request once if it failed, but only once. Otherwise, it should throw an exception.

How would I write gevent code with at pool that supports re-running HTTP requests once if they fail?

Could this approach work?

import requests
import gevent
from gevent.pool import Pool

pool = Pool(10)

def do_request(id):
    r = requests.get('http://example.com/%u' % id)
    if not r.status_code == 200:
        raise RuntimeError(id)

def spawn_greenlet(id, is_retry=False):
    if not is_retry:
        g = gevent.spawn(id)
        g.link_exception(retry_once)
    else:
        g = pool.spawn(id)
        g.link_exception(raise_exception)
    return g

def retry_once(greenlet):
    return spawn_greenlet(greenlet.exception.args[0])

def raise_exception(greenlet):
    if greenlet.exception:
        raise greenlet.exception
    raise RuntimeError('Unknown error in greenlet processing.')


greenlets = pool.map(spawn_greenlet, [1, 2, 3, 4, 5])
gevent.joinall(greenlets)

• Ist there a cleaner way to obtain the argument of the greenlet function than via exception arguments?
• Is there a possibility that the joinall(greenlets) methods returns after an exception occurs inside do_request but before the retry_once event handler is called?
• Is there a cleaner way to restart a greenlet with the same arguments, so I wouldn't need the is_retry kwarg at spawn_greenlet ?
• As far as I understand this, gevent.joinall(greenlets) only joins the greenlets returned by map. When there's an exception, is the original greenlet replaced with the new one returned by retry_once ? If not, does processing continue even though the additional greenlets are still running? How could I wait for all greenlets to finish in that case?

Gevent docs are very scarce and there seem to be no other resources in the web documenting this, even though this is a fairly common use case. Therefore I don't consider this a too localized question.

Answer 1

Don't use spawn/link/link_exception for retrying things. Just use normal Python:

def do_something_with_retry(*args):
    try:
      return do_something(*args)
    except Exception:
      return do_something(*args)

Also, gevent.pool.Pool.map automatically spawns greenlet within given pool, you don't have to do it.

pool = Pool(10)
pool.map(do_something_with_retry, [1, 2, 3])

Now, you only need to implement do_something() , which can be normal Python/requests code:

def do_something(*args):
    return requests.get('http://gevent.org')

Have fun!