End API Pagination Loop in Python

Question

I am trying to build a Python script where I retrieve all the invoices from Chargebee through their API. I am, however, coming up a bit short as I do not know how to properly end the loop once it has reached the last page. My current solution consists of inserting a break when the URL is the same as the last page. Unfortunately, this is not very sustainable as I have to update it every time the last next_offset value changes.

My current code looks like this:

import requests
import json
import pandas as pd
from requests.auth import HTTPBasicAuth
from pandas import json_normalize 

invoices = []
customers = []

def GetInvoices():
    url = 'https://company.chargebee.com/api/v2/invoices?limit=100&offset='
    url1 = 'https://company.chargebee.com/api/v2/invoices?limit=100&offset='
    while url:
        print('----')
        print('Requesting', url)
        response = requests.get(url, auth=HTTPBasicAuth('APIKEY','pass'))
        data = response.json()

        invoices.extend(data['list'])

        if url == 'https://company.chargebee.com/api/v2/invoices?limit=100&offset=["1507705200000","78206775"]':
            df = json_normalize(invoices)
            df.to_csv('InvoicesUS.csv')
            print('----')
            print('Invoice list has been exported!')
            break

        else:
            url = url1 + data['next_offset']

I am very new to Python, so there might be a lot of unnecessary things in the script. It is working, but I would like to find a way to page through the API and close the loop once the last page has been appended.

Thank you!

Answer 1

Found a solution to the problem by using try/except. The code now looks like this:

import requests
import json
import pandas as pd
from requests.auth import HTTPBasicAuth
from pandas import json_normalize

invoices = []

url = 'https://company.chargebee.com/api/v2/invoices?limit=100&offset='
url1 = 'https://company.chargebee.com/api/v2/invoices?limit=100&offset='

while url:
    try:
        print('----')
        print('Requesting', url)
        response = requests.get(url, auth=HTTPBasicAuth('APIKEY','pass'))
        data = response.json()

        invoices.extend(data['list'])
        
        url = url1 + data['next_offset']

    except KeyError:
        df = json_normalize(invoicesUS)
        print('----')
        print('Invoice list has been exported!')
        break