How do I handle exceptions in Python

Question

This answered my question, problem solved. You may delete this post.

RETRIES = 10

id = None
session = requests.Session()

for attempt in range(1, RETRIES + 1):
    response = session.get(url)
    soup = BeautifulSoup(r.text, "lxml")

    element = soup.find('a', class_="class", id=True)
    if element is None:
        print("Attempt {attempt}. Element not found".format(attempt=attempt))
        continue
    else:
        id = element["id"]
        break

print(id)

This answered my question, problem solved. You may delete this post.

Answer 1

You can apply the "look before you leap" ( LBYL ) principle and check the result of find() - it would return None if an element was not found. You can then put the thing into a loop and exit when you have a value, also safeguarding yourself with a loop counter limit:

RETRIES = 10

id = None
session = requests.Session()

for attempt in range(1, RETRIES + 1):
    response = session.get(url)
    soup = BeautifulSoup(r.text, "lxml")

    element = soup.find('a', class_="class", id=True)
    if element is None:
        print("Attempt {attempt}. Element not found".format(attempt=attempt))
        continue
    else:
        id = element["id"]
        break

print(id)

Couple notes:

• id=True was set to find only elements with the id element present. You could also do an equivalent with a CSS selector soup.select_one("a.class[id]")
• Session() helps to improve performance when issuing requests to the same host multiple times. See more at Session Objects

Answer 2

If all you want to do is make that same request a second time, you could do something like this:

import requests
from bs4 import BeautifulSoup

def find_data(url):
    found_data = False
    while not found_data:
        r = requests.get(url)
        soup = BeautifulSoup(r.text, "lxml")
        try:
            id = soup.find('a', class_="class").get('id')
            found_data = True
        except:
            pass

This puts you at risk of an infinite loop if the data really aren't there. You can do this to avoid that infinite loop:

import requests
from bs4 import BeautifulSoup

def find_data(url, attempts_before_fail=3):
    found_data = False
    while not found_data:
        r = requests.get(url)
        soup = BeautifulSoup(r.text, "lxml")
        try:
            id = soup.find('a', class_="class").get('id')
            found_data = True
        except:
            attempts_before_fail -= 1
            if attempts_before_fail == 0:
                raise ValueError("couldn't find data after all.")