Using tweepy to get unique tweets

Question

I am trying to get a corpus of Tweets using a number of search terms. One issue I am having is that it is not being able to get unique tweets. That is, retweets.

Is there a way to remove these beforehand without doing any text processing?

What I've got now:

 api=tweepy.API(auth)
 for search in hashtags:
     for tweet in  tweepy.Cursor(api.search,q=search,count=1000,lang="en").items(): 
         text=repr(tweet.text.encode("utf-8"))  
         out.write(text+"\n")

Answer 1

You can add " -filter:retweets" to your query to only get original tweets. Maybe not the prettiest solution, but it works.

api=tweepy.API(auth)
for search in hashtags:
    for tweet in  tweepy.Cursor(api.search,q=search+" -filter:retweets",count=1000,lang="en").items(): 
        text=repr(tweet.text.encode("utf-8"))  
        out.write(text+"\n")