如何在将列表中的每个值传递到 API 并在每个列表列表后暂停时迭代列表列表？

Question

I have a list that looks like this:

lst = [1,2,3, etc]

I can successfully iterate over this list and pass the IDs into an API. However the API breaks after certain point in regards to number of Ids I am passing.

I went ahead and broke the list into chunks of 50 using this code:

def chunks(l, n):
    # For item i in a range that is a length of l,
    for i in range(0, len(l), n):
        # Create an index range for l of n items:
        yield l[i:i+n]

a = list(chunks(lst, 50))

The a variable is list of list that looks like this:

[[1,2,3][4,5,6]]

How do I iterate over the list of list, passing individual Ids into the API code (below), while pausing the code for 10seconds after every chunk?

In addition, if the API breaks, is there anyway I can force the API to continue rest of the code ? Not a requirement for now, so this is optional.

API code:

lst1=[]
for i in lst:
    #print (row)
    url = 'url/Id={}'.format(i)
    r = requests.get(url).text
    lst1.append(xml_data1)
    #time.sleep(1)
    print(xml_data1)

Thank you in advance.

Answer 1

You can use a double for loop with use of the sleep function after execution of the inner loop. Here's an example:

lst1 = []
for chunk in a:
    for id in chunk:
        url = 'url/Id={}'.format(id)
        xml_data = requests.get(url).text
        lst1.append(xml_data)
     sleep(10) # will pause execution for 10 seconds

To answer your second question about how to continue execution if the API throws an error, you can use try/except . Code that might throw an error goes in the try block and if an error is thrown, execution moves to the except block, which specifies how to proceed. If you wrap your code in a function this can be made a bit easier.

I find it helpful to track the number of times you've tried calling for a specific id with a keyword argument that can be used for exponential backoff -- where you wait for longer amounts of time depending on how many times you've tried.

Here is an example building on the above code:

def get_data_on_id(id, try_num=1):
    url = 'url/Id={}'.format(id)
    try:
        return requests.get(url).text
    except:
        sleep(2**try_num) # exponential backoff
        return get_data_on_id(id, try_num=try_num + 1)

lst1 = []
for chunk in a:
    for id in chunk:
        xml_data = get_data_on_id(id)
        lst1.append(xml_data)
    sleep(10)

The code above is largely the same as the code before it, but the request logic is wrapped in the get_data_on_id function, which contains the try/except blocks. If an error is thrown when requesting the url , the function will wait for 2^try_num seconds and then try again.

You might also want to except specific errors and handle them differently (or just to be explicit so you only except errors you are prepared to handle). Here are the docs on try/except : https://docs.python.org/3/tutorial/errors.html