IndexError：使用BeautifulSoup从网页中提取表

Question

I am trying to extract a table (with columns - flight, carrier, origin, date, scheduled, estimated, status) containing flight arrival information from an airport website , but I got the error below:

  IndexError Traceback (most recent call last)
  <ipython-input-39-2f7369a95ba9> in <module>()
        6  for cl in cols:
        7  dv = cl.findAll('div')
  ----> 8  if 'col-xs-12 col-sm-6' in dv[0]['class']:
        9  flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
       10 print(flight, carrier, origin, date, scheduled, estimated, status)
  IndexError: list index out of range

I have combed stackoverflow for solution but could not find solution. Here is my code:

  # import libraries
  import urllib3
  import requests
  from bs4 import BeautifulSoup

  # query the website and return the html to the variable ‘page’
  page = requests.get("https://www.aucklandairport.co.nz/flights").text

  soup = BeautifulSoup(page)
  tbody = soup.findAll('tbody')
  for tb in tbody:
    rows = tb.findAll('tr')
    for tr in rows:
      cols = tr.findAll('td')
        for cl in cols:
          dv = cl.findAll('div')
          if 'col-xs-12 col-sm-6' in dv[0]['class']:
             flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
             print(flight, carrier, origin, date, scheduled, estimated, status)

Thanks for your contribution.

Answer 1

The problem is that the first td under tr don't have the div , that's why dv will return empty. change your code to:

# import libraries
import requests
from bs4 import BeautifulSoup

# query the website and return the html to the variable ‘page’
page = requests.get("https://www.aucklandairport.co.nz/flights").text

soup = BeautifulSoup(page)
tbody = soup.find('tbody')
rows = tbody.findAll('tr',{'class':'flight-toggle'}) #find tr whose class = flight-toggle
for tr in rows:
    cols = tr.findAll('td',class_=lambda x: x != 'logo') # find td whose class!=logo (exclude the first td)
    dv0 = cols[0].find('div').findAll('div') #flight, carrier, origin under second td
    flight, carrier, origin = [c.text.strip() for c in dv0]
    dv1 = cols[1].find('div').findAll('div') #date, schedule under third td
    date, scheduled = [c.text.strip() for c in dv1]
    dv2 = cols[2].find('div').findAll('div') #estimated, statusunder fouth td
    estimated, status = [c.text.strip() for c in dv2[1:]] # exclude the first div 
    print(flight, carrier, origin, date, scheduled, estimated, status)

This will print out:

(u'EK406', u'', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'QF8762', u'EK406', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'EK434', u'', u'Dubai / Brisbane', u'18 Nov', u'02:45pm', u'02:49pm', u'Processing')
...

Answer 2

You can peel the same apple in different ways. Here is another way you can achieve the same.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.aucklandairport.co.nz/flights")
soup = BeautifulSoup(response.text,"lxml")
table = soup.select(".flights-table")[0]
for items in table.select("tr.flight-toggle"):
    data = ' '.join([' '.join(item.text.split()) for item in items.select("td")])
    print(data.strip())

Partial result:

QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
AA7377 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
BA7421 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
CZ7575 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing