推文抓取 - 如何衡量推文强度？

Question

I am looking for a method to get information of a "trend" regarding some hashtag/key word on Twitter. Let`s say I want to measure how often the hashtag/key word "Python" is tweeted in time. For instance, today, "Python" is tweeted on average every 1 minute but yesterday it was tweeted on average every 2 minutes.

I have tried various options but I am always bouncing off the twitter API limitations, ie if I try to download all tweets for a hashtag during the last (for example) day, only a certain franction of the tweets is downloaded (via tweepy.cursor).

Do you have any ideas / script examples of achieving similar results? Libraries or guides to recommend? I did not find any help searching on the internet. Thank you.

Answer 1

You should check twint repository.

• Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
• Fast initial setup;
• Can be used anonymously and without Twitter sign up;

here is a sample code:

import twint


def scrapeData(search):
    c = twint.Config()

    c.Search = search

    c.Since = '2021-03-05 00:00:00'
    c.Until = '2021-03-06 00:00:00'
    c.Pandas = True
    c.Store_csv = True
    c.Hide_output = True
    c.Output = f'{search}.csv'
    c.Limit = 10  # number of tweets want to fetch

    print(f"\n#### Scraping from {c.Since} to {c.Until}")
    twint.run.Search(c)

    print("\n#### Preview: ")
    print(twint.storage.panda.Tweets_df.head())


if __name__ == "__main__":
    scrapeData(search="python")

Answer 2

Try a library called: GetOldTweets or GetOldTweets3

Twitter Search, and by extension its API, are not meant to be an exhaustive source of tweets. The Twitter Streaming API places a limit of just one week on how far back tweets can be extracted from that match the input parameters. So in order to extract all historical tweets relevant to a set of search parameters for analysis, the Twitter Official API needs to be bypassed and custom libraries that mimic the Twitter Search Engine need to be used.