The number of search results on twitter web pages and the difference in results in tweepy

Question

import tweepy
import csv #Import csv

CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_KEY = ''
ACCESS_SECRET = ''

auth = tweepy.auth.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)

api = tweepy.API(auth, wait_on_rate_limit=False)
# Open/Create a file to append data
csvFile = open('BB.csv', 'a')
#Use csv Writer
csvWriter = csv.writer(csvFile)
#max_tweets = 100

for tweet in tweepy.Cursor(api.search, 
                q="bondai beach", 
                lang="en").items():
#Write a row to the csv file/ I use encode utf-8
csvWriter.writerow([tweet.text.encode('utf-8')])
print(tweet.text)
csvFile.close()

I'm working on importing Twitter data into Python using the Tweepy. Is there a way to resolve the difference between the number of results retrieved from a web page and the number retrieved by api? For example, if you search for bondai beach, you will get a lot of results on the web page, but if you use the API to load it, there will be only 3 results. What's even more odd is that there are a lot of results that can be retrieved from the API depending on the keyword. Using the API to retrieve data for an eiffel tower yields a lot of results it is almost more than 1000 until I stop the code by force. The above is my code.

Answer 1

Is there a way to resolve the difference between the number of results retrieved from a web page and the number retrieved by api?

Twitter's internal serach API is likely different from their public API, so unfortunately this is only possible if you were to hijack their search API directly (which is likely against their Terms of Service).

What's even more odd is that there are a lot of results that can be retrieved from the API depending on the keyword.

I'm not sure what is odd about this, but it might be a good idea to implement either a maximum, or some kind of pagination implementation.