如何从收集的数据集中删除转发

Question

i have a collected dataset of tweets in python (jupyter notebook). but there are many duplicate tweets. how can i remove these programmaticaly with python (jupyter notebook)

csvFile = open('ua.csv', 'a')
csvWriter = csv.writer(csvFile)

search_words = "corona"
date_since = "2020-10-13"

new_search = search_words + " -filter:retweets"
new_search

for tweet in tweepy.Cursor(api.search,q=search_words,count=100,
                           lang="id",
                           since=date_since).items():
    print (tweet.created_at, tweet.text)
    csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])

Answer 1

While you're iterating through the list of tweets, you could keep a list of tweets in a set, and check if you've already written that tweet.

tweet_set = set() # store tweet ids you've already seen before
for tweet in tweepy.Cursor(api.search,q=search_words,count=100,
                           lang="id",
                           since=date_since).items():

    if tweet.id not in tweet_set:
        print (tweet.created_at, tweet.text)
        csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])

        tweet_set.add(tweet.id) # update the set of tweets