Getting all the pages from an API

Question

This is something I struggle with, or whenever I do it it seems to be messy. I'm going to ask the question in a very generic way as it's not a single problem I'm really trying to solve.

I have an API that I want to consume some data from, eg via:

def get_api_results(page)
  results = HTTParty.get("api.api.com?page=#{page}")
end

When I call it I can retrieve a total.

results["total"] = 237

The API limits the number of records I can retrieve in one call, say 20. So I need to call it a few more times.

I want to do something like the following, ideally breaking it into pieces so I can use things like delayed_job..etc

def get_all_api_pages
  results = get_api_results(1)
  total = get_api_results(1)["total"]

  until page*20 > total do |p|
    results += get_api_results(p)
  end
end

I always feel like I'm writing rubbish whenever I try and solve this (and I've tried to solve it in a number of ways). The above, for example, leaves me at the mercy of an error with the API, which knocks out all my collected results if I hit an error at any point.

Wondering if there is just a generally good, clean way of dealing with this situation.

Answer 1

I don't think you can have that much cleaner...because you only receive the total once you called the API. Have you tried to build your own enum for this. It encapsulates the ugly part. Here is a bit of sample code with a "mocked" API:

class AllRecords
  PER_PAGE = 50

  def each
    return enum_for(:each) unless block_given?
    current_page = 0
    total = nil
    while total.nil? || current_page * PER_PAGE < total
      current_page += 1
      page = load_page(current_page)
      total = page[:total]
      page[:items].each do |item|
        yield(item)
      end
    end
  end

  private

  def load_page(page)
    if page == 5
      {items: Array.new(37) { rand(100) }, total: 237}
    else
      {items: Array.new(50) { rand(100) }, total: 237}
    end
  end
end

AllRecords.new.each.each_with_index do |item, index|
  p index
end

You can surely clean that out a bit but i think that this is nice because it does not collect all the items first.