Difficulty with Twitter API Search

Question

I'm new to python Twitter API .
I have set up auth with twitter, and I tried the search which works fine.

t = twitter.Twitter(auth=auth)    
tt = t.search.tweets(q="stackoverflow",rpp=50)

This worked fine and I can read the output.

But if I try:

for page in range(1,6):
    tt = t.search.tweets(q="stackoverflow", rpp=100, page=page)

This crashes with the error:

details: {"errors":[{"code":44,"message":"page parameter is invalid"}]}

I looked on the Twitter page and it seems that the page parameter is ok as an optional parameter: https://dev.twitter.com/docs/api/1/get/search .

Alternatively, how do I get more search results if the rate is limited to 100 results per request. Is there another work around it?

Answer 1

You should use the max_id value to iterate through the results. Make a first request to the Twitter API , read max_id from the result and send it in your next request.

From the docs :

To use max_id correctly, an application's first request to a timeline endpoint should only specify a count. When processing this and subsequent responses, keep track of the lowest ID received. This ID should be passed as the value of the max_id parameter for the next request, which will only return Tweets with IDs lower than or equal to the value of the max_id parameter. Note that since the max_id parameter is inclusive, the Tweet with the matching ID will actually be returned again, as shown in the following image:

Use Twython :

from twython import Twython

twitter = Twython(APP_KEY, access_token=ACCESS_TOKEN)

# Initial request
result = twitter.search(q='stackoverflow', count=100)

# Get max_id of results
next_max_id = result['search_metadata']['next_results'].split('max_id=')[1].split('&')[0]
# Use max_id in next request 
result = twitter.search(q='stackoverflow', count=100, max_id=next_max_id)