Ruby on Rails - Issues with Iterating Through Twitter Gem Search Results

Question

I've been using the Twitter Streaming API for a while with no problems, but recently switched over to using the REST and am a bit confused.

I'm trying to iterate through the results of tweets using #vietnam. After reading the documentation , it appears I should do this using the max_id.

max_id = 70978401017678643100000000000000 #very big number to start out with
i = 1
2.times do
  puts '----------SEARCHING WITH MAX_ID: ' + max_id.to_s
  $twitter.search('#vietnam', result_type: "recent", lang: 'en', geocode: "14.05832400,108.27719900,100000mi", count: 100, max_id: max_id).each do |tweet|
    puts "#{i} - tweet.id: " + tweet.id.to_s
    i += 1
    max_id = tweet.id if tweet.id < max_id
  end
end

Here are the results from the terminal:

----------SEARCHING WITH MAX_ID: 70978401017678643100000000000000
1 - tweet.id: 713034427027361794
2 - tweet.id: 713030624164982785
3 - tweet.id: 713027195015413760
4 - tweet.id: 713022173246582784
5 - tweet.id: 713021690989641728
6 - tweet.id: 713013561396764672
7 - tweet.id: 713010847166177280
8 - tweet.id: 712997640930197505
...
341 - tweet.id: 709784010176786432
----------SEARCHING WITH MAX_ID: 709784010176786432
342 - tweet.id: 709784010176786432

The geocode in the search query has a 100,000 mile radius from the center of Vietnam to ensure that it my results are not being limited by location. The problem is that during the second iteration, the results only return 1 tweet, the max_id from the last result.

I know there are more 341 tweets with #vietnam in English, as verified by Twitter's advanced search functionality: https://twitter.com/search-advanced?lang=en

What is wrong with my iteration technique?

Answer 1

Twitter's REST API only returns results from the past 7 days, it's possible only 341 tweets with #vietnam have been made in the last week. The advanced search has results from much further back.

Your method seems sound, try a less restrictive search with a simple or popular term that's been used thousands of times over the past week and see if you get more results.

Answer 2

Is there any reason you are performing the Search two separate times?

You are finding the lowest (oldest) tweet id in the first iteration, and searching for results older than that in the second iteration (which don't exist). Perhaps I misunderstood?