如何从下拉列表中抓取数据

Question

我正在刮擦下面的待售汽车网页，并且我想从下拉列表中提取所有汽车制造商（例如，奥迪，宝马，福特，爱特斯），但我不知道如何：访问： https：// www .sgcarmart.com / used_cars / listing.php？MOD = Foton＆RPG = 40＆VEH = 0＆AVL = 2

html代码如下：

<a href="listing.php?MOD=Foton&amp;RPG=40&amp;VEH=0&amp;AVL=2" style="background-color:#0066AA; color:#FFFFFF;">Foton</a><a href="listing.php?MOD=Alfa&amp;RPG=40&amp;VEH=0&amp;AVL=2">Alfa</a><a href="listing.php?MOD=Alfa+Romeo&amp;RPG=40&amp;VEH=0&amp;AVL=2">Alfa Romeo</a><a href="listing.php?MOD=Aston+Martin&amp;RPG=40&amp;VEH=0&amp;AVL=2">Aston Martin</a><a href="listing.php?MOD=Audi&amp;RPG=40&amp;VEH=0&amp;AVL=2">Audi</a><a href="listing.php?MOD=Austin&amp;RPG=40&amp;VEH=0&amp;AVL=2">Austin</a><a href="listing.php?MOD=Bentley&amp;RPG=40&amp;VEH=0&amp;AVL=2">Bentley</a><a href="listing.php?MOD=BMW&amp;RPG=40&amp;VEH=0&amp;AVL=2">BMW</a><a href="listing.php?MOD=Chery&amp;RPG=40&amp;VEH=0&amp;AVL=2">Chery</a><a href="listing.php?MOD=Chevrolet&amp;RPG=40&amp;VEH=0&amp;AVL=2">Chevrolet</a><a href="listing.php?MOD=Chrysler&amp;RPG=40&amp;VEH=0&amp;AVL=2">Chrysler</a><a href="listing.php?MOD=Citroen&amp;RPG=40&amp;VEH=0&amp;AVL=2">Citroen</a><a href="listing.php?MOD=Daihatsu&amp;RPG=40&amp;VEH=0&amp;AVL=2">Daihatsu</a><a href="listing.php?MOD=Datsun&amp;RPG=40&amp;VEH=0&amp;AVL=2">Datsun</a><a href="listing.php?MOD=Ferrari&amp;RPG=40&amp;VEH=0&amp;AVL=2">Ferrari</a><a href="listing.php?MOD=Fiat&amp;RPG=40&amp;VEH=0&amp;AVL=2">Fiat</a><a href="listing.php?MOD=Ford&amp;RPG=40&amp;VEH=0&amp;AVL=2">Ford</a><a href="listing.php?MOD=Golden+Dragon&amp;RPG=40&amp;VEH=0&amp;AVL=2">Golden Dragon</a><a href="listing.php?MOD=Higer&amp;RPG=40&amp;VEH=0&amp;AVL=2">Higer</a><a href="listing.php?MOD=Hino&amp;RPG=40&amp;VEH=0&amp;AVL=2">Hino</a><a href="listing.php?MOD=Honda&amp;RPG=40&amp;VEH=0&amp;AVL=2">Honda</a><a href="listing.php?MOD=Hummer&amp;RPG=40&amp;VEH=0&amp;AVL=2">Hummer</a><a href="listing.php?MOD=Hyundai&amp;RPG=40&amp;VEH=0&amp;AVL=2">Hyundai</a><a href="listing.php?MOD=Infiniti&amp;RPG=40&amp;VEH=0&amp;AVL=2">Infiniti</a><a href="listing.php?MOD=Isuzu&amp;RPG=40&amp;VEH=0&amp;AVL=2">Isuzu</a><a href="listing.php?MOD=Iveco&amp;RPG=40&amp;VEH=0&amp;AVL=2">Iveco</a><a href="listing.php?MOD=JAC&amp;RPG=40&amp;VEH=0&amp;AVL=2">JAC</a><a href="listing.php?MOD=Jaguar&amp;RPG=40&amp;VEH=0&amp;AVL=2">Jaguar</a><a href="listing.php?MOD=Jeep&amp;RPG=40&amp;VEH=0&amp;AVL=2">Jeep</a><a href="listing.php?MOD=Jensen&amp;RPG=40&amp;VEH=0&amp;AVL=2">Jensen</a><a href="listing.php?MOD=Joylong&amp;RPG=40&amp;VEH=0&amp;AVL=2">Joylong</a><a href="listing.php?MOD=Kia&amp;RPG=40&amp;VEH=0&amp;AVL=2">Kia</a><a href="listing.php?MOD=King+Long&amp;RPG=40&amp;VEH=0&amp;AVL=2">King Long</a><a href="listing.php?MOD=Lamborghini&amp;RPG=40&amp;VEH=0&amp;AVL=2">Lamborghini</a><a href="listing.php?MOD=Land+Rover&amp;RPG=40&amp;VEH=0&amp;AVL=2">Land Rover</a><a href="listing.php?MOD=Lexus&amp;RPG=40&amp;VEH=0&amp;AVL=2">Lexus</a><a href="listing.php?MOD=Lotus&amp;RPG=40&amp;VEH=0&amp;AVL=2">Lotus</a><a href="listing.php?MOD=MAN&amp;RPG=40&amp;VEH=0&amp;AVL=2">MAN</a><a href="listing.php?MOD=Maserati&amp;RPG=40&amp;VEH=0&amp;AVL=2">Maserati</a><a href="listing.php?MOD=Maxus&amp;RPG=40&amp;VEH=0&amp;AVL=2">Maxus</a><a href="listing.php?MOD=Maybach&amp;RPG=40&amp;VEH=0&amp;AVL=2">Maybach</a><a href="listing.php?MOD=Mazda&amp;RPG=40&amp;VEH=0&amp;AVL=2">Mazda</a><a href="listing.php?MOD=McLaren&amp;RPG=40&amp;VEH=0&amp;AVL=2">McLaren</a><a href="listing.php?MOD=Mercedes-Benz&amp;RPG=40&amp;VEH=0&amp;AVL=2">Mercedes-Benz</a>

我尝试了以下代码：

r = requests.get('https://www.sgcarmart.com/used_cars/listing.php?RPG=40&MOD=audi&VEH=0&RGD=&AVL=2&ORD=')
soup = BeautifulSoup(r.text, 'html.parser')
res_2 = soup.find_all('div', attrs={'id': 'make_attach_menu_child'})
makes = re.findall(r'<a href=\"listing.php\?MOD\=.+?\>(.+)', str(res_2))

Answer 1

您可以从以下网址开始请求： https : //www.sgcarmart.com/used_cars/listing.php? MOD = Foton & RPG =40& VEH =0& AVL =2

import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")

然后通过BeautifulSoup解析它：

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')

生成列表位于div中，其类为：make_child dropdown_child因此，我们找到了它：

lis = soup.find('div',attrs={'class':'make_child dropdown_child'})

现在，每个品牌都放在一个标签中，因此我遍历了lis并将文本附加到列表中：

makes = []
for i in lis.findAll('a') :
     makes.append (i.text)

然后打印列表，将得到以下输出：

编译整个代码：

import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')
lis = soup.find('div',attrs={'class':'make_child dropdown_child'})
makes = []
for i in lis.findAll('a') :
    makes.append (i.text)