Python catch timeout and repeat request

Question

I'm trying to use the Xively API with python to update a datastream but occasionally I get a 504 error which seems to end my script.

How can I catch that error and more importantly delay and try again so the script can keep going and upload my data a minute or so later?

Here's the block where I'm doing the uploading.

    # Upload to Xivity
    api = xively.XivelyAPIClient("[MY_API_KEY")
    feed = api.feeds.get([MY_DATASTREAM_ID])
    now = datetime.datetime.utcnow()
    feed.datastreams = [xively.Datastream(id='temps', current_value=tempF, at=now)]
    feed.update()

And here's the error I see logged when my script fails:

Traceback (most recent call last):
 File "C:\[My Path] \ [My_script].py", line 39, in <module>
   feed = api.feeds.get([MY_DATASTREAM_ID])
 File "C:\Python34\lib\site-packages\xively_python-0.1.0_rc2-py3.4.egg\xively\managers.py", >line 268, in get
   response.raise_for_status()
 File "C:\Python34\lib\site-packages\requests-2.3.0-py3.4.egg\requests\models.py", line 795, >in raise_for_status
   raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out

Thanks,

PS I've replaced my personal info with [MY_INFO] but obviously the correct data appears in my code.

Answer 1

I usually use a decorator for this:

from functools import wraps
from requests.exceptions import HTTPError
import time

def retry(func):
    """ Call `func` with a retry.

    If `func` raises an HTTPError, sleep for 5 seconds
    and then retry.

    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            ret = func(*args, **kwargs)
        except HTTPError:
            time.sleep(5)
            ret = func(*args, **kwargs)
        return ret
    return wrapper

Or, if you want to retry more than once:

def retry_multi(max_retries):
    """ Retry a function `max_retries` times. """
    def retry(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            num_retries = 0 
            while num_retries <= max_retries:
                try:
                    ret = func(*args, **kwargs)
                    break
                except HTTPError:
                    if num_retries == max_retries:
                        raise
                    num_retries += 1
                    time.sleep(5)
            return ret 
        return wrapper
    return retry

Then put your code in a function like this

#@retry
@retry_multi(5) # retry 5 times before giving up.
def do_call():
    # Upload to Xivity
    api = xively.XivelyAPIClient("[MY_API_KEY")
    feed = api.feeds.get([MY_DATASTREAM_ID])
    now = datetime.datetime.utcnow()
    feed.datastreams = [xively.Datastream(id='temps', current_value=tempF, at=now)]
    feed.update()

Answer 2

You could throw in a try/except statement in a loop that has a sleep timer for however long you want to wait between tries. Something like this:

import time

# Upload to Xivity
api = xively.XivelyAPIClient("[MY_API_KEY")
feed = api.feeds.get([MY_DATASTREAM_ID])
now = datetime.datetime.utcnow()
feed.datastreams = [xively.Datastream(id='temps', current_value=tempF, at=now)]

### Try loop
feed_updated = False
while feed_updated == False:
    try: 
        feed.update()
        feed_updated=True
    except: time.sleep(60)

EDIT As Dano pointed out, it would be better to have a more specific except statement.

### Try loop
feed_updated = False
while feed_updated == False:
    try: 
        feed.update()
        feed_updated=True
    except HTTPError: time.sleep(60) ##Just needs more time.
    except: ## Otherwise, you have bigger fish to fry
        print "Unidentified Error"
        ## In such a case, there has been some other kind of error. 
        ## Not sure how you prefer this handled. 
        ## Maybe update a log file and quit, or have some kind of notification, 
        ## depending on how you are monitoring it. 

Edit a general except statement.

### Try loop
feed_updated = False
feed_update_count = 0
while feed_updated == False:
    try: 
        feed.update()
        feed_updated=True
    except: 
        time.sleep(60)
        feed_update_count +=1 ## Updates counter

    if feed_update_count >= 60:    ## This will exit the loop if it tries too many times
        feed.update()              ## By running the feed.update() once more,
                                   ## it should print whatever error it is hitting, and crash