Tweepy (Twitter API) Not Returning all Search Results

Question

I'm using the search feature with Tweepy for Twitter and for some reason the search results are limited to 15. Here is my code

results=api.search(q="Football",rpp=1000)

for result in results:
    print "%s" %(clNormalizeString(result.text))

print len(results)

and only 15 results are returned. Does it have something to do with different pages of results or something?

Answer 1

The question is more about Twitter API instead of tweepy itself.

According to the documentation , count parameter defines:

The number of tweets to return per page, up to a maximum of 100. Defaults to 15. This was formerly the "rpp" parameter in the old Search API.

FYI, you can use tweepy.Cursor to get paginated results, like this:

import tweepy


auth = tweepy.OAuthHandler(..., ...)
auth.set_access_token(..., ...)

api = tweepy.API(auth)
for tweet in tweepy.Cursor(api.search,
                           q="google",
                           count=100,
                           result_type="recent",
                           include_entities=True,
                           lang="en").items():
    print tweet.created_at, tweet.text

See also: https://github.com/tweepy/tweepy/issues/197 .

Hope that helps.

Answer 2

Here's a minimal working example (once you replace the fake keys with real ones).

import tweepy
from math import ceil

def get_authorization():

    info = {"consumer_key": "A7055154EEFAKE31BD4E4F3B01F679",
            "consumer_secret": "C8578274816FAEBEB3B5054447B6046F34B41F52",
            "access_token": "15225728-3TtzidHIj6HCLBsaKX7fNpuEUGWHHmQJGeF",
            "access_secret": "61E3D5BD2E1341FFD235DF58B9E2FC2C22BADAD0"}

    auth = tweepy.OAuthHandler(info['consumer_key'], info['consumer_secret'])
    auth.set_access_token(info['access_token'], info['access_secret'])
    return auth


def get_tweets(query, n):
    _max_queries = 100  # arbitrarily chosen value
    api = tweepy.API(get_authorization())

    tweets = tweet_batch = api.search(q=query, count=n)
    ct = 1
    while len(tweets) < n and ct < _max_queries:
        print(len(tweets))
        tweet_batch = api.search(q=query, 
                                 count=n - len(tweets),
                                 max_id=tweet_batch.max_id)
        tweets.extend(tweet_batch)
        ct += 1
    return tweets

Note: I did try using a for loop, but the twitter api sometimes returns fewer than 100 results (despite being asked for 100, and 100 being available). I'm not sure why this is, but that's the reason why I didn't include a check to break the loop if tweet_batch is empty -- you may want to add such a check yourself as there is a query rate limit .

Another Note: You can avoid hitting the rate limit by invoking wait_on_rate_limit=True like so

        api = tweepy.API(get_authorization(), wait_on_rate_limit=True)