Handling rate limit exceptions with Cursor in tweepy

Question

I'm trying to find the correct way to handle rate limits when iterating through a list of followers using the Cursor object. Here is what I'm trying:

while True:
    try:
        for follower in tweepy.Cursor(api.followers, id=root_usr).items():
            print(follower.id)
    except tweepy.TweepError:
        # hit rate limit, sleep for 15 minutes
        print('Rate limited. Sleeping for 15 minutes.')
        time.sleep(15 * 60 + 15)
        continue
    except StopIteration:
        break

This is probably incorrect, since an exception will make the for loop start from the beginning again. What's the right way to iterate through all of root_usr 's followers while also handling the rate limit problems?

Answer 1

You can construct the Cursor instance on a variable and call next(cursor) inside a try-catch. You can handle StopIteration and TweepError's that way.

Like

cursor = tweepy.Cursor(api.followers, id=root_usr).items()
# or tweepy.Cursor(api.followers, id=root_usr).items().iter()
while True:
    try:
        follower = next(cursor)
        print(follower.id)
    except StopIteration:
        break
    except tweepy.TweepError:
        # Handle this, sleep, log, etc.
        pass

Answer 2

There's an example under the code snippets section of the documentation that suggests making a wrapper function to deal with error handling.

def limit_handled(cursor):
    while True:
        try:
            yield next(cursor)
        except tweepy.RateLimitError:
            time.sleep(15 * 60)
        except StopIteration:
            print("Done")
            break


for follower in limit_handled(tweepy.Cursor(api.followers, id=root_usr).items()):
    print(follower.id)

Also, I suggest setting count to the max, tweepy.Cursor(api.followers, id=root_usr, count=200).items() , to make the most of each API call.