How to fork and merge a REST API call that takes a list of id's as query param?

Question

I have to call a third party GET API which takes a comma seperated list of ids and returns a JSON response. Like so:

def call(sysid)
   # RestClient GET  https://test.com?ids=sysid.join(',')&....
   # parse response and returns
   # rescue - any error handling
end

RestClient

But sysid is an input and can sometimes have 300+ elements and due to the limitation of GET, my api call errors out (probably because URL length becomes too big)

How can I efficiently send API calls (in small batches) & merge results back. In doing that: -

• should I use any Parallelism
  • if one of the batch call fails, may be add retries
    • if it still fail, stop the whole process & return an error from call()

Is there a good pattern in Ruby for this type of problem?

Answer 1

I am not aware of good ruby pattern but we can solve issue by below code

def call(sysid)
  fetch_next = true
  response = {status: true, result: []}
  while fetch_next
    param_value_id = sysid[0..n-1]
    while sysid.present?
      begin
        call API with FARADAY/RestClient with its params as param_value_id.join(',')
        if success
          parse response and append in responses result
        else
          fetch_next = false
          response[:status] = false
          response[:result] = []
        end
      rescue Exception => e
        Rails.logger.warn "Exception raised due to #{e}"
        fetch_next = false
        response[:status] = false
        response[:result] = []
      end
    end
    sysid = sysid.drop(n)
    if sysid.empty?
      fetch_next = false
    end
  end
  response
end

EXPLANATION

1. setting fetch_next to true and response hash to be returned from call method.
2. till fetch_next is true , we'll fetch n ids from sysid
3. till sysid is there, we'll call third party api with intended params
4. if response is success , we append it in result key of response hash
5. if there is failure or an exception , we'll set fetch_next to false , status key of response to false and empty result key
6. In the end we drop n values from sysid array and will set fetch_next to false if sysid is empty.

you can return response object or send particular response if status is false

This is pseudo code but i think it might solve your problem.

Thank you.