How to break for loop while handling API error so you don't lose all your data

Question

I have a for loop that is continually passing a parameter to an API. I am taking the responses from the API call and appending them to 3 lists. Once the for loop is completed, I am creating a pandas dataframe which I am then write to a .csv file.

The entire loop takes like 20 min to run and I just got an error 10 min in, so I lost all the data before it was able to write to a dataframe and then .csv file.

How would you handle this situation better?

# Create empty lists for data
email = []
is_disposable_address = []
mailbox_verification = []

for address in emails_to_validate:

    parameters = {"address": address}
    response = requests.get(url, params=parameters,
                auth=("api", "key"))

    email.append(address)
    is_disposable_address.append(response.json()["is_disposable_address"])
    mailbox_verification.append(response.json()["mailbox_verification"])

# Combine lists into dictionary and create dataframe
data = {"email_address": email,
        "is_disposable_address": is_disposable_address,
        "mailbox_verification": mailbox_verification}

dF = pd.DataFrame(data)

# Export to csv
dF.to_csv("emails.csv", index=False)

Answer 1

The simplest solution is to just wrap the loop in a try/catch block like so:

try:
    for address in emails_to_validate:

        parameters = {"address": address}
        response = requests.get(url, params=parameters,
            auth=("api", "key"))

        email.append(address)
        is_disposable_address.append(response.json()["is_disposable_address"])
        mailbox_verification.append(response.json()["mailbox_verification"])

except Exception as e:
    print('An exception has occurred.  Giving up on loop:')
    print(e)

Execution will then continue to the subsequent part of your code.

You could also put the Try / Except block inside the loop if you want to continue working on subsequent list items.

See the docs on exception handling for more detail on how to use try/except and do more nuanced conditionals on error type.