IndexError：使用BeautifulSoup从网页中提取表

Question

我正在尝试从机场网站提取包含航班抵达信息的表（带有列-航班，承运人，出发地，日期，预定，估计，状态），但出现以下错误：

  IndexError Traceback (most recent call last)
  <ipython-input-39-2f7369a95ba9> in <module>()
        6  for cl in cols:
        7  dv = cl.findAll('div')
  ----> 8  if 'col-xs-12 col-sm-6' in dv[0]['class']:
        9  flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
       10 print(flight, carrier, origin, date, scheduled, estimated, status)
  IndexError: list index out of range

我已经对stackoverflow进行了梳理，但是找不到解决方案。 这是我的代码：

  # import libraries
  import urllib3
  import requests
  from bs4 import BeautifulSoup

  # query the website and return the html to the variable ‘page’
  page = requests.get("https://www.aucklandairport.co.nz/flights").text

  soup = BeautifulSoup(page)
  tbody = soup.findAll('tbody')
  for tb in tbody:
    rows = tb.findAll('tr')
    for tr in rows:
      cols = tr.findAll('td')
        for cl in cols:
          dv = cl.findAll('div')
          if 'col-xs-12 col-sm-6' in dv[0]['class']:
             flight, carrier, origin, date, scheduled, estimated, status = [c.text for c in dv]
             print(flight, carrier, origin, date, scheduled, estimated, status)

感谢您的贡献。

Answer 1

问题是，第一个td在tr没有div ，这就是为什么dv将返回空。 将您的代码更改为：

# import libraries
import requests
from bs4 import BeautifulSoup

# query the website and return the html to the variable ‘page’
page = requests.get("https://www.aucklandairport.co.nz/flights").text

soup = BeautifulSoup(page)
tbody = soup.find('tbody')
rows = tbody.findAll('tr',{'class':'flight-toggle'}) #find tr whose class = flight-toggle
for tr in rows:
    cols = tr.findAll('td',class_=lambda x: x != 'logo') # find td whose class!=logo (exclude the first td)
    dv0 = cols[0].find('div').findAll('div') #flight, carrier, origin under second td
    flight, carrier, origin = [c.text.strip() for c in dv0]
    dv1 = cols[1].find('div').findAll('div') #date, schedule under third td
    date, scheduled = [c.text.strip() for c in dv1]
    dv2 = cols[2].find('div').findAll('div') #estimated, statusunder fouth td
    estimated, status = [c.text.strip() for c in dv2[1:]] # exclude the first div 
    print(flight, carrier, origin, date, scheduled, estimated, status)

这将打印出：

(u'EK406', u'', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'QF8762', u'EK406', u'Dubai / Melbourne', u'18 Nov', u'01:55pm', u'02:47pm', u'Processing')
(u'EK434', u'', u'Dubai / Brisbane', u'18 Nov', u'02:45pm', u'02:49pm', u'Processing')
...

Answer 2

您可以用不同的方法去皮。 这是您可以达到相同目的的另一种方法。

import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.aucklandairport.co.nz/flights")
soup = BeautifulSoup(response.text,"lxml")
table = soup.select(".flights-table")[0]
for items in table.select("tr.flight-toggle"):
    data = ' '.join([' '.join(item.text.split()) for item in items.select("td")])
    print(data.strip())

部分结果：

QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
AA7377 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
BA7421 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing
CZ7575 QF145 Sydney 18 Nov 05:05pm Est 05:32pm Processing