python recursive decorator

Question

I have a function that returns a list of dictionaries from an API call. The API can return max 100 records per request. Records are stored in pages, each page containing 100 records.

def call_api(page):
    return list_of_records

what I need is a decorator that can re-run this function every time it returns 100 records with the page argument =+ 1.

something like:

if len(list_of_records) == 100:
   call_api(page +=1)

the final output should be one single list with all the records

Answer 1

Defining a decorator that recursively calls a function is certainly possible.

Here is how such a decorator might look in your case:

import functools

def with_pages(func):
    @functools.wraps(func)

    def wrapper(page):
        list_of_records = []
        partial_list = func(page)
        list_of_records.extend(partial_list)
        while len(partial_list) == 100:  # there is probably a better condition to check if you are at the last page (e.g. next_page URI in api reply)
            page += 1
            partial_list = func(page)
            list_of_records.extend(partial_list)
        return list_of_records
    return wrapper

@with_pages
def call_api(page):
    # api interaction creating list_of_records_on_a_page
    return list_of_records_on_a_page

---

Alternatively, you can adapt call_api such that you can call it recursively.

Something like:

def call_api(page, list_of_records=[]):
    partial_list = ... # the records you get from the api
    list_of_records.extend(partial_list)
    # often api provide the url for the next page or some more reliable info
    # about whether or not you are on the last page, i.e. testing for the length
    # of the reply is probably not the best strategy
    if len(partial_list) == 100:
        # time.sleep(0.2)  # just to not end up with to many requests in a short period of time
        call_api(page + 1, list_of_records)
    return list_of_records

Simply calling call_api(1) will be enough to get all pages.