Nested loop in Python when using multiple parameters in API calls

Question

I have a program to call an API, return the JSON data and write it to a CSV. The program loops through a list of entities as the first parameter in the API call, but also now needs to loop through a second parameter set (start and end times in epoch) because the API has a max of pulling a week of data at a time.

Example:

API call: ex.com/api/entity/timecards?start_time=1531306800&end_time=1531846800&pretty=1

So I need to loop through all of the entities, and then loop through an entire year's worth of data, a week at a time.

code example so far for the API call function:

 def callAPI(entities):
        for j in range(len(entities)):

           locnum = entities[j][:5]
           locnumv = entities[j]

           startTime =
           endTime = 

           url = "http://ex.com/api/entity/" + entity[j] + "/timecards?start_time=" + startTime + "&end_time=" + endTime

           querystring = {"pretty":"1"}

           headers = {
            'Api-Key': ""
            }

           r = requests.request("GET", url, headers=headers, params=querystring)

           d = r.json()

The program then goes on to write the data to rows in a CSV, which is all successful when tested with looping through the entities with static time parameters.

So I just need to figure out how would I create another nested for loop to loop through the start time/end time + 518400 seconds (6 days instead of 7 to be safe) and factor in a timeout since this is effectively going to be 20,000+ API calls by the time it's all said and done?

Answer 1

First off, you can just do:

for entity in entities:

instead of:

for j in range(len(entities)):

and then use entity instead of entities[j]

When it comes to looping through your epoch times. You will have to set your start time and then set your end time to start_time + 540000 inside of another loop:

   start_time = 1531306800
   i = 0
   while True:
       if i != 0:
           start_time = end_time

       end_time = start_time + 540000

       url = "http://ex.com/api/entity/" + entity + "/timecards?start_time=" + start_time + "&end_time=" + end_time

       querystring = {"pretty":"1"}

       headers = {'Api-Key': ""}

       try:
           r = requests.request("GET", url, headers=headers, params=querystring)
       except:
           break

       d = r.json()

Basically, you are going to loop through all of the epoch times until the request fails. Once it does, you will exit the loop and go to your next entity. The new entity's url will start at the same epoch time as the entity before it and so on.

I hope that helps!

Answer 2

First of all, because you are just using j for getting the current entity, you could replace for j in range(len(entities)) by for entity in entities , it reads better. As for the question, you could just use an inner for loop to iterate over each week. The whole code will be:

def callAPI(entities):
  for entity in entities:
    locnum = entity[:5]
    locnumv = entity # This is redundant

    START = 1531306800  # starting time, 1 year ago
    END = START + 31536000  # final time, e.g. the current time
    TIME_STEP = 518400  # 1 day, 1 week, 1 month

    for start_time in range(START, END, TIME_STEP):
      end_time = start_time + TIME_STEP - 1 # substract 1 for preventing overlap of times

      url = "http://ex.com/api/entity/%s/timecards?start_time=%d&end_time=%d" % (entity, start_time, end_time)

      querystring = {"pretty":"1"}

      headers = {'Api-Key': ""}

      try:
          r = requests.request("GET", url, headers=headers, params=querystring)
      except:
          break

      d = r.json()

      # Do something with the data

I hope this can help you!!