Python、Beautiful Soup、WebScraping、Pandas、Dataframe

Question

复杂的 Beautiful Soup 查询

我开始熟悉 Beautiful Soup 和 Pandas 的 Dataframe，但我似乎无法将两者结合起来。

import urllib.request
from bs4 import BeautifulSoup
import pandas as pd


connection = urllib.request.urlopen('http://www.carfolio.com/specifications/models/?man=557')
soup = BeautifulSoup(connection, "html.parser", from_encoding='utf-7')

soup.decode('utf-7','ignore')

href_tag = soup.find_all(span="detail")
for href_tag in soup.body.stripped_strings:
    print(str(href_tag.encode('utf-7')))

最终，我的目标是抓取每辆车并创建一个包含相关信息（“细节”）的数据框，例如马力、扭矩、重量等。我只是不知道如何“抓取”细节.

我环顾四周，有一些例子，但大多数都没有访问“缩写标题”谢谢

Answer 1

如果您可以为列表中的每辆车发出额外请求，那么这里是一个如何获取汽车特征的示例工作演示：

>>> import requests
>>> from bs4 import BeautifulSoup
>>> 
>>> soup = BeautifulSoup(requests.get("http://www.carfolio.com/specifications/models/car/?car=427691").content)
>>> for item in soup.select("div.summary dl dt"):
...     print(item.get_text(strip=True), item.find_next_sibling("dd").get_text(strip=True))
... 
(u'What body style?', u'hatchback with 4/5 seats')
(u'How long?', u'3973mm')
(u'How heavy?', u'1110kg')
(u'What size engine?', u'1 litre, 999cm3')
(u'How many cylinders?', u'3, Straight')
(u'How much power?', u'95PS/ 94bhp/ 70kW@ 5000-5500rpm')
(u'How much torque?', u'160Nm/ 118ft.lb/ 16.3kgm@ 1500-3500rpm')
(u'How quick?', u'0-100km/h: 10.9s')
(u'How fast?', u'186km/h, 116mph')
(u'How economical?', u'5.0/3.7/4.2 l/100km urban/extra-urban/combined')
(u'Whatcarbon dioxide emissions?', u'97.0CO2g/km')

Answer 2

对于每辆车，您可以从 li 标签获取信息：

>>>from bs4 import BeautifulSoup

>>>url="""<li class="detail"><a href="car/?car=214027" class="addstable"><span class="automobile"><span class="manufacturer" title="Manufacturer">Manexall</span>, <span class="modelyear" title="Model year">1921 <abbr title="model year">MY</abbr></span> </span></a><span class="detail"> <abbr title="front engine, rear wheel drive">FR</abbr> 1143 <abbr title="cubic centimetres">cm<sup>3</sup></abbr> 13.2 <abbr title="Pferdestärke">PS</abbr> 13 <abbr title="brake horsepower">bhp</abbr> 9.7 <abbr title="kilowatts">kW</abbr> 363 <abbr title="kilograms">kg</abbr></span></li>"""

>>>soup = BeautifulSoup(url)
>>>[data.get_text() for data in soup.select('li')]
['Manexall, 1921 MY \n FR 1143 cm3 13.2 PS 13 bhp 9.7 kW 363 kg']