使用 HTTParty 和 Ruby on Rails 使用分页的 Github API 数据

Question

I'm building a web scraper for Github's repository data and storing specific repo attributes in a local database. I'm currently running into an issue pulling data beyond their one page (100 records) limit.

Here's my api call and method to extract the appropriate data:

require 'active_interaction'
require 'json'
class GitHubGet < ActiveInteraction::Base
  def execute
    response = HTTParty.get(process_path)
    # extract_github_data(response)
  end

  def extract_github_data(response)
    parsed_response = JSON.parse(response.body)
    result = []
    parsed_response["items"].each do |item|
      if item["updated_at"] > 1.day.ago
        result << {
          name: item["name"],
          owner: item["owner"]["login"],
          url: item["url"],
          stars: item["stargazers_count"]
        }
      end
    end
    puts result 
  end 

  private

  def process_path
    "https://api.github.com/search/repositories?q=license:mit+license:apache-2.0+license:gpl+license:lgpl+stars:1..2000+fork:false&per_page=100"
  end

end 

Any help on how to pull in more than one page of data would be greatly appreciated! Thanks!

Answer 1

响应对象中的标头有一个键链接，带有到下一页的 url。