Accommodating Twitter's rate limit in Python's Twython

Question

I'm pulling data through Twitter's REST API using Twython. I want the code to automatically rest as long as it needs to when it's reached the Twitter rate limit, then begin querying again.

Here's the code, which takes a list of Twitter IDs and adds their followers'IDs to the list:

for user in first_ids:
    try:
        followers = twitter.get_followers_ids(user_id=user, count=600)
        for individual in followers['ids']:    
            if individual not in ids:
                ids.append(individual)
    except TwythonRateLimitError as error:
        remainder = float(twitter.get_lastfunction_header(header='x-rate-limit-reset')) - time.time()
        time.sleep(remainder)
        continue

When I run it I get the following error: "Connection aborted. Error 10054: An existing connection was forcibly closed by the remote host"

What does the error mean? I imagine it's related to Twitter's rate limit -- is there another way around it?

Answer 1

you're leaving the connection open while your program sleeps, try closing it manually and then connecting again after the sleep timeout. Something like:

    except TwythonRateLimitError as error:
        remainder = float(twitter.get_lastfunction_header(header='x-rate-limit-reset')) - time.time()
        twitter.disconnect()
        time.sleep(remainder)
        twitter = Twython(APP_KEY, APP_SECRET,OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
        continue

Answer 2

if you are using REST api you can use the same solution deleting the api instead of using .disconnect() simply use

del twitter

instead of

twitter.disconnect()

i had the same problem and it worked for me