celery + eventlet = 100% CPU usage

Question

We are using celery to get flights data from different travel agencies, every request takes ~20-30 seconds(most agencies require request sequence - authorize, send request, poll for results).

Normal celery task looks like this:

from eventlet.green import urllib2, time 
def get_results(attr, **kwargs): 
    search, provider, minprice = attr 
    data = XXX # prepared data 
    host = urljoin(MAIN_URL, "RPCService/Flights_SearchStart") 
    req = urllib2.Request(host, data, {'Content-Type': 'text/xml'}) 
    try: 
        response_stream = urllib2.urlopen(req) 
    except urllib2.URLError as e: 
        return [search, None] 
    response = response_stream.read() 
    rsp_host = urljoin(MAIN_URL, "RPCService/FlightSearchResults_Get") 
    rsp_req = urllib2.Request(rsp_host, response, {'Content-Type': 
'text/xml'}) 
    ready = False 
    sleeptime = 1 
    rsp_response = '' 
    while not ready: 
        time.sleep(sleeptime) 
        try: 
            rsp_response_stream = urllib2.urlopen(rsp_req) 
        except urllib2.URLError as e: 
            log.error('go2see: results fetch failed for %s IOError %s'% 
(search.id, str(e))) 
        else: 
            rsp_response = rsp_response_stream.read() 
            try: 
                rsp = parseString(rsp_response) 
            except ExpatError as e: 
                return [search, None] 
            else: 
                ready = rsp.getElementsByTagName('SearchResultEx') 
[0].getElementsByTagName('IsReady')[0].firstChild.data 
                ready = (ready == 'true') 
        sleeptime += 1 
        if sleeptime > 10: 
            return [search, None] 
    hash = "%032x" % random.getrandbits(128) 
    open(RESULT_TMP_FOLDER+hash, 'w+').write(rsp_response) 
   # call to parser 
    parse_agent_results.apply_async(queue='parsers', args=[__name__, 
search, provider, hash]) 

This tasks are run in eventlet pool with concurency 300, prefetch_multiplier = 1 , broker_limit = 300 When ~100-200 task are fetched from queue - CPU usage raises up to 100% ( whole CPU core is used) and task fetching from queue is performed with delays.

Could you please point on possible issues - blocking operations( eventlet ALARM DETECTOR gives no exceptions ), wrong architecture or whatever.

Answer 1

Sorry for late response.

Thing i would try first in such situation is to turn off Eventlet completely in both Celery and your code, use process or OS thread model. 300 threads or even processes is not that much load for OS scheduler (although you may lack memory to run many processes). So i would try it and see if CPU load drops dramatically. If it does not, then problem is in your code and Eventlet can't magically fix it. If it does drop, however, we would need to investigate the issue closer.

If bug still persists, please, report it via any of these ways:

• https://bitbucket.org/which_linden/eventlet/issues/new
• https://github.com/eventlet/eventlet/issues/new
• email to eventletdev@lists.secondlife.com

Answer 2

A problem occurs if you fire 200 requests to a server, responses could be delayed and therefore urllib.urlopen will hang.

Another thing i noticed: If an URLError is raised, the program stays in the while loop until sleeptime is greater than 10. So an URLError error will let this script sleep for 55 sec (1+2+3.. etc)