繁体 English 中英

When Web Scraping data doesn't go as plan with python, BeautifulSoup and csv [closed]

[英]When Web Scraping data doesn't go as plan with python, BeautifulSoup and csv [closed]

原文 2022-01-24 16:48:00 8 0 python/ csv/ web-scraping/ beautifulsoup/ request

查看抓取此 URL ** https://www.airport-data.com/manuf/Reims.html* * 的详细信息时，很容易抓取尾号字段的数值。

飞机注册后，会获得一个注册号。 这个数字显示在飞机的尾部。 当一架飞机在尾号 0008 下注册时。该号码是其合法注册号，就像汽车牌照一样。 该代码不尊重此类尾号。

所以8而不是0008和10而不是0010和Mar-1956而不是03-1956 。
下面的代码很好地抓取了这个网站，但完全不尊重被抓取数据的细节。 可以在代码中构建适当的抓取逻辑吗？

import requests, csv
from bs4 import BeautifulSoup
from urllib.request import Request

headers =   {
        'accept':'*/*',
        'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
        }

url = 'https://www.airport-data.com/manuf/Reims.html'

with open('Beech.csv', "w", encoding="utf-8-sig") as f:
    writer = csv.writer(f)
    writer.writerow(["Tail_Number", "Year_Maker_Model", "C_N","Engines", "Seats", "Location"])

    while True:
        print(url)
        html = requests.get(url, headers=headers)
        soup = BeautifulSoup(html.text, 'html.parser')

        # Go throught table = tbody and extract the data under the 'td' tag
        for row in soup.select('table tbody tr'):
            writer.writerow([c.text if c.text else '' for c in row.select('td')])

        # If more than one page then iterate through all of them        
        if soup.select_one('li.active + li a'):
            url = soup.select_one('li.active + li a')['href']
        else:
            break