Looping through Twitter search query results - Ruby on Rails gem

Question

I'm building a reporting tool to help a social media agency collect stats on the accounts they manage. I am using the Twitter gem .

I would like to collect the number of @mentions an account received that week.

Since there are lots of accounts (around 70, let's say), I would like the tool to use one user context to gather the data rather than repeatedly logging in and out as different accounts.

To my knowledge this means I can't use the mentions_timeline request as that requires specific user context.

Instead I am doing this:

twitter_mentions = twitter(twitter_token, twitter_secret).search("@{account.twitter_handle}", options = { since: from_date, until: to_date, count: 100 }).count

def from_date
  9.days.ago.strftime("%Y-%m-%d")
end

def to_date
  2.days.ago.strftime("%Y-%m-%d")
end

def twitter(token, secret)
  Twitter::REST::Client.new do |config|
    config.consumer_key        = ENV["twitter_api_key"]
    config.consumer_secret     = ENV["twitter_api_secret"]
    config.access_token        = token
    config.access_token_secret = secret
  end
end

The only problem is that the max number of tweets returned is limited to 100. I am aware from the documentation that I can use max_id to perhaps make several seperate calls which I can then sum up, however I am having trouble constructing this elegantly in Ruby.

Any help appreciated.

Answer 1

I can't remember exactly how I went about this, but it is possible to 'listen' to a twitter search live with the twitter live-streaming API.

I've done this with one of my projects using the tweetstream gem. Which receives tweets in real time and runs a block of code on each one in turn. In theory this would be able to drop rows in a database, as long as it's not returning many results a second.

Here's some relevant code from my project ( https://github.com/AJFaraday/Text-to-music/blob/master/scripts/twitter-search.rb ):

consumer_key, consumer_secret, oauth_token, oauth_token_secret = get_twitter_auth(config)

TweetStream.configure do |config|
  config.consumer_key       = consumer_key
  config.consumer_secret    = consumer_secret
  config.oauth_token        = oauth_token
  config.oauth_token_secret = oauth_token_secret
  config.auth_method        = :oauth
end

# Set up new TweetStream client
ts = TweetStream::Client.new

ts.track("@username") do |status|
  puts "[#{status.user.screen_name}] - #{status.source}"
  puts status.text
end

(the get_twitter_auth method is in https://github.com/AJFaraday/Text-to-music/blob/master/lib/twitter-auth.rb , which stores the initial twitter authorisation in the twitter.yml file.)

Hope this helps.