How to stop celery task while it is being executed and continue the execution after some time?

Question

I have the following celery task(simplified), which interacts with Twitter API.

@app.task
def get_followers(screen_name, **kwargs):
    cursor = kwargs.get('cursor', -1)
    followers = kwargs.get('followers', [])
    while True:
        response = twitter_api.call('followers', 'ids', screen_name=screen_name, cursor=cursor)
        if response.status_code == '429': # RATE LIMIT EXCEEDED
            # do something here

        cursor = response.json()['next_cursor']
        if cursor == 0: # we're done
            break
    return followers

I want to be able to pause task for some time when the rate limit is hit, and resume the execution from the point where it left off. (Or throw an error and retry the task, passing in additional kwargs). How can this be accomplished?

Answer 1

You can just retry your task when you catch a 429 error code :

@app.task(bind=True)
def get_followers(self, screen_name, **kwargs):
    cursor = kwargs.get('cursor', -1)
    followers = kwargs.get('followers', [])
    while True:
        response = twitter_api.call('followers', 'ids', screen_name=screen_name, cursor=cursor)
        if response.status_code == '429': 
            # RATE LIMIT EXCEEDED
            self.retry(countdown=15*60)

        cursor = response.json()['next_cursor']
        if cursor == 0: # we're done
            break
    return followers

Note that I added bind=True in your task decorator and self as parameter in the definition of your task to be able to do self.retry when you get a 429.

in retry use the argument countdown to say when you want the task to be retried (in seconds). Here I chose 15min (twitter API rate limits)

You can find more infos about retrying in the celery documentation :

http://docs.celeryproject.org/en/latest/userguide/tasks.html#retrying