Web scraping error

Question

I am extracting some data from the website in the code below, but I run into some problem with the duration in this line duration = tr.select('td.duration')[0].contents[0].strip() , which throws the exception below.Please how can I fix that line thank you in order to extract the duration data. I have searched similar questions on SO but they do not quite answer my question.

# import needed libraries
from mechanize import Browser
from bs4 import BeautifulSoup
import csv

br = Browser()

# Ignore robots.txt
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Chrome')]

# Retrieve the home page
br.open('http://fahrplan.sbb.ch/bin/query.exe/en')
br.select_form(nr=6)

br.form["REQ0JourneyStopsS0G"] = 'Eisenstadt'  # Origin train station (From)
br.form["REQ0JourneyStopsZ0G"] = 'sarajevo'  # Destination train station (To)
br.form["REQ0JourneyTime"] = '5:30'  # Search Time
br.form["date"] = '18.01.17'  # Search Date

# Get the search results
br.submit()

# get the response from mechanize Browser
soup = BeautifulSoup(br.response().read(), 'lxml', from_encoding="utf-8")
trs = soup.select('table.hfs_overview tr')

# scrape the contents of the table to csv (This is not complete as I cannot write the duration column to the csv)
with open('out.csv', 'w') as f:
    for tr in trs:
        locations = tr.select('td.location')
        if len(locations) > 0:
            location = locations[0].contents[0].strip()
            prefix = tr.select('td.prefix')[0].contents[0].strip()
            time = tr.select('td.time')[0].contents[0].strip()
            duration = tr.select('td.duration')[0].contents[0].strip()
            f.write("{},{},{},{}\n".format(location.encode('utf-8'), prefix, time, duration))

Traceback (most recent call last):
  File "C:/.../tester.py", line 204, in <module>
    duration = tr.select('td.duration')[0].contents[0].strip()
IndexError: list index out of range

Process finished with exit code 1

Answer 1

Either tr.select('td.duration') is a list of length zero, or tr.select('td.duration')[0].contents is a list of length zero. You need to guard against these possibilities somehow. One approach is to use conditionals.

durations = tr.select('td.duration')
if len(durations) == 0:
    print("oops! There aren't any durations.")
else:
    contents = durations[0].contents
    if len(contents) == 0:
        print("oops! There aren't any contents.")
    else:
        duration = contents[0].strip()
        #rest of code goes here

Or perhaps you'd like to simply ignore TRs that don't fit your expected model, in which case a try-catch might suffice.

try:
    duration = tr.select('td.duration')[0].contents[0].strip()
except IndexError:
    print("Oops! tr didn't have expected tds and/or contents.")
    continue
#rest of code goes here