有什么方法可以加快使用tweepy下载推文的python代码吗？

Question

Here is the code that i am using for the purpose .For each user request it's taking too long time to download all the tweets.What are some ways to speed up the execution time.The idea is to use tweet analytics in real time as the user visits the website.I am new to python so any help would be appreciated .

import tweepy #https://github.com/tweepy/tweepy


#Twitter API credentials
consumer_key = ".."
consumer_secret = ".."
access_key = ".."
access_secret = ".."


def get_all_tweets(screen_name):
    #Twitter only allows access to a users most recent 3240 tweets with this method

    #authorize twitter, initialize tweepy
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_key, access_secret)
    api = tweepy.API(auth)

    #initialize a list to hold all the tweepy Tweets
    alltweets = []  

    #make initial request for most recent tweets (200 is the maximum allowed count)
    new_tweets = api.user_timeline(screen_name = screen_name,count=200)

    #save most recent tweets
    alltweets.extend(new_tweets)

    #save the id of the oldest tweet less one
    oldest = alltweets[-1].id - 1

    #keep grabbing tweets until there are no tweets left to grab
    while len(new_tweets) > 0:
        print ("getting tweets before %s".format(oldest))

        #all subsiquent requests use the max_id param to prevent duplicates
        new_tweets = api.user_timeline(screen_name = screen_name,count=200,max_id=oldest)

        #save most recent tweets
        alltweets.extend(new_tweets)

        #update the id of the oldest tweet less one
        oldest = alltweets[-1].id - 1

        print ("...%s tweets downloaded so far".format(len(alltweets)))

    #transform the tweepy tweets into a 2D array that will populate the csv 
    outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode("utf-8")] for tweet in alltweets]
    return outtweets

Answer 1

One way to make your solution faster would be to make some cache.

When you've downloaded all the tweets for a screen name, save them locally, for instance as [twitter_screen_name].json

Then edit your function to check for your cache files. If it doesn't exist, create it empty. Then load it, refresh only what needs to, and save back your json cache file.

This way, when a user visits, you'll download only the diff with twitter. This will be much faster for the regularly consulted screen names.

Then you could add something for auto clearing the cache - a simple CRON that removes files with last-accessed META older than n days for instance.