[英]How to scrape data from dropdown list
我正在刮擦下面的待售汽車網頁,並且我想從下拉列表中提取所有汽車制造商(例如,奧迪,寶馬,福特,愛特斯),但我不知道如何:訪問: https:// www .sgcarmart.com / used_cars / listing.php?MOD = Foton&RPG = 40&VEH = 0&AVL = 2
html代碼如下:
<a href="listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2" style="background-color:#0066AA; color:#FFFFFF;">Foton</a><a href="listing.php?MOD=Alfa&RPG=40&VEH=0&AVL=2">Alfa</a><a href="listing.php?MOD=Alfa+Romeo&RPG=40&VEH=0&AVL=2">Alfa Romeo</a><a href="listing.php?MOD=Aston+Martin&RPG=40&VEH=0&AVL=2">Aston Martin</a><a href="listing.php?MOD=Audi&RPG=40&VEH=0&AVL=2">Audi</a><a href="listing.php?MOD=Austin&RPG=40&VEH=0&AVL=2">Austin</a><a href="listing.php?MOD=Bentley&RPG=40&VEH=0&AVL=2">Bentley</a><a href="listing.php?MOD=BMW&RPG=40&VEH=0&AVL=2">BMW</a><a href="listing.php?MOD=Chery&RPG=40&VEH=0&AVL=2">Chery</a><a href="listing.php?MOD=Chevrolet&RPG=40&VEH=0&AVL=2">Chevrolet</a><a href="listing.php?MOD=Chrysler&RPG=40&VEH=0&AVL=2">Chrysler</a><a href="listing.php?MOD=Citroen&RPG=40&VEH=0&AVL=2">Citroen</a><a href="listing.php?MOD=Daihatsu&RPG=40&VEH=0&AVL=2">Daihatsu</a><a href="listing.php?MOD=Datsun&RPG=40&VEH=0&AVL=2">Datsun</a><a href="listing.php?MOD=Ferrari&RPG=40&VEH=0&AVL=2">Ferrari</a><a href="listing.php?MOD=Fiat&RPG=40&VEH=0&AVL=2">Fiat</a><a href="listing.php?MOD=Ford&RPG=40&VEH=0&AVL=2">Ford</a><a href="listing.php?MOD=Golden+Dragon&RPG=40&VEH=0&AVL=2">Golden Dragon</a><a href="listing.php?MOD=Higer&RPG=40&VEH=0&AVL=2">Higer</a><a href="listing.php?MOD=Hino&RPG=40&VEH=0&AVL=2">Hino</a><a href="listing.php?MOD=Honda&RPG=40&VEH=0&AVL=2">Honda</a><a href="listing.php?MOD=Hummer&RPG=40&VEH=0&AVL=2">Hummer</a><a href="listing.php?MOD=Hyundai&RPG=40&VEH=0&AVL=2">Hyundai</a><a href="listing.php?MOD=Infiniti&RPG=40&VEH=0&AVL=2">Infiniti</a><a href="listing.php?MOD=Isuzu&RPG=40&VEH=0&AVL=2">Isuzu</a><a href="listing.php?MOD=Iveco&RPG=40&VEH=0&AVL=2">Iveco</a><a href="listing.php?MOD=JAC&RPG=40&VEH=0&AVL=2">JAC</a><a href="listing.php?MOD=Jaguar&RPG=40&VEH=0&AVL=2">Jaguar</a><a href="listing.php?MOD=Jeep&RPG=40&VEH=0&AVL=2">Jeep</a><a href="listing.php?MOD=Jensen&RPG=40&VEH=0&AVL=2">Jensen</a><a href="listing.php?MOD=Joylong&RPG=40&VEH=0&AVL=2">Joylong</a><a href="listing.php?MOD=Kia&RPG=40&VEH=0&AVL=2">Kia</a><a href="listing.php?MOD=King+Long&RPG=40&VEH=0&AVL=2">King Long</a><a href="listing.php?MOD=Lamborghini&RPG=40&VEH=0&AVL=2">Lamborghini</a><a href="listing.php?MOD=Land+Rover&RPG=40&VEH=0&AVL=2">Land Rover</a><a href="listing.php?MOD=Lexus&RPG=40&VEH=0&AVL=2">Lexus</a><a href="listing.php?MOD=Lotus&RPG=40&VEH=0&AVL=2">Lotus</a><a href="listing.php?MOD=MAN&RPG=40&VEH=0&AVL=2">MAN</a><a href="listing.php?MOD=Maserati&RPG=40&VEH=0&AVL=2">Maserati</a><a href="listing.php?MOD=Maxus&RPG=40&VEH=0&AVL=2">Maxus</a><a href="listing.php?MOD=Maybach&RPG=40&VEH=0&AVL=2">Maybach</a><a href="listing.php?MOD=Mazda&RPG=40&VEH=0&AVL=2">Mazda</a><a href="listing.php?MOD=McLaren&RPG=40&VEH=0&AVL=2">McLaren</a><a href="listing.php?MOD=Mercedes-Benz&RPG=40&VEH=0&AVL=2">Mercedes-Benz</a>
我嘗試了以下代碼:
r = requests.get('https://www.sgcarmart.com/used_cars/listing.php?RPG=40&MOD=audi&VEH=0&RGD=&AVL=2&ORD=')
soup = BeautifulSoup(r.text, 'html.parser')
res_2 = soup.find_all('div', attrs={'id': 'make_attach_menu_child'})
makes = re.findall(r'<a href=\"listing.php\?MOD\=.+?\>(.+)', str(res_2))
您可以從以下網址開始請求: https : //www.sgcarmart.com/used_cars/listing.php? MOD = Foton & RPG =40& VEH =0& AVL =2
import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")
然后通過BeautifulSoup解析它:
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')
生成列表位於div中,其類為:make_child dropdown_child因此,我們找到了它:
lis = soup.find('div',attrs={'class':'make_child dropdown_child'})
現在,每個品牌都放在一個標簽中,因此我遍歷了lis並將文本附加到列表中:
makes = []
for i in lis.findAll('a') :
makes.append (i.text)
編譯整個代碼:
import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')
lis = soup.find('div',attrs={'class':'make_child dropdown_child'})
makes = []
for i in lis.findAll('a') :
makes.append (i.text)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.