[英]How to scrape data from dropdown list
I'm scraping the below sale car web page and I want to extract all car makes from a drop down list (for example, audi, bmw, ford, ets) and I don't know how : visit: https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2 我正在刮擦下面的待售汽车网页,并且我想从下拉列表中提取所有汽车制造商(例如,奥迪,宝马,福特,爱特斯),但我不知道如何:访问: https:// www .sgcarmart.com / used_cars / listing.php?MOD = Foton&RPG = 40&VEH = 0&AVL = 2
the html code is like: html代码如下:
<a href="listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2" style="background-color:#0066AA; color:#FFFFFF;">Foton</a><a href="listing.php?MOD=Alfa&RPG=40&VEH=0&AVL=2">Alfa</a><a href="listing.php?MOD=Alfa+Romeo&RPG=40&VEH=0&AVL=2">Alfa Romeo</a><a href="listing.php?MOD=Aston+Martin&RPG=40&VEH=0&AVL=2">Aston Martin</a><a href="listing.php?MOD=Audi&RPG=40&VEH=0&AVL=2">Audi</a><a href="listing.php?MOD=Austin&RPG=40&VEH=0&AVL=2">Austin</a><a href="listing.php?MOD=Bentley&RPG=40&VEH=0&AVL=2">Bentley</a><a href="listing.php?MOD=BMW&RPG=40&VEH=0&AVL=2">BMW</a><a href="listing.php?MOD=Chery&RPG=40&VEH=0&AVL=2">Chery</a><a href="listing.php?MOD=Chevrolet&RPG=40&VEH=0&AVL=2">Chevrolet</a><a href="listing.php?MOD=Chrysler&RPG=40&VEH=0&AVL=2">Chrysler</a><a href="listing.php?MOD=Citroen&RPG=40&VEH=0&AVL=2">Citroen</a><a href="listing.php?MOD=Daihatsu&RPG=40&VEH=0&AVL=2">Daihatsu</a><a href="listing.php?MOD=Datsun&RPG=40&VEH=0&AVL=2">Datsun</a><a href="listing.php?MOD=Ferrari&RPG=40&VEH=0&AVL=2">Ferrari</a><a href="listing.php?MOD=Fiat&RPG=40&VEH=0&AVL=2">Fiat</a><a href="listing.php?MOD=Ford&RPG=40&VEH=0&AVL=2">Ford</a><a href="listing.php?MOD=Golden+Dragon&RPG=40&VEH=0&AVL=2">Golden Dragon</a><a href="listing.php?MOD=Higer&RPG=40&VEH=0&AVL=2">Higer</a><a href="listing.php?MOD=Hino&RPG=40&VEH=0&AVL=2">Hino</a><a href="listing.php?MOD=Honda&RPG=40&VEH=0&AVL=2">Honda</a><a href="listing.php?MOD=Hummer&RPG=40&VEH=0&AVL=2">Hummer</a><a href="listing.php?MOD=Hyundai&RPG=40&VEH=0&AVL=2">Hyundai</a><a href="listing.php?MOD=Infiniti&RPG=40&VEH=0&AVL=2">Infiniti</a><a href="listing.php?MOD=Isuzu&RPG=40&VEH=0&AVL=2">Isuzu</a><a href="listing.php?MOD=Iveco&RPG=40&VEH=0&AVL=2">Iveco</a><a href="listing.php?MOD=JAC&RPG=40&VEH=0&AVL=2">JAC</a><a href="listing.php?MOD=Jaguar&RPG=40&VEH=0&AVL=2">Jaguar</a><a href="listing.php?MOD=Jeep&RPG=40&VEH=0&AVL=2">Jeep</a><a href="listing.php?MOD=Jensen&RPG=40&VEH=0&AVL=2">Jensen</a><a href="listing.php?MOD=Joylong&RPG=40&VEH=0&AVL=2">Joylong</a><a href="listing.php?MOD=Kia&RPG=40&VEH=0&AVL=2">Kia</a><a href="listing.php?MOD=King+Long&RPG=40&VEH=0&AVL=2">King Long</a><a href="listing.php?MOD=Lamborghini&RPG=40&VEH=0&AVL=2">Lamborghini</a><a href="listing.php?MOD=Land+Rover&RPG=40&VEH=0&AVL=2">Land Rover</a><a href="listing.php?MOD=Lexus&RPG=40&VEH=0&AVL=2">Lexus</a><a href="listing.php?MOD=Lotus&RPG=40&VEH=0&AVL=2">Lotus</a><a href="listing.php?MOD=MAN&RPG=40&VEH=0&AVL=2">MAN</a><a href="listing.php?MOD=Maserati&RPG=40&VEH=0&AVL=2">Maserati</a><a href="listing.php?MOD=Maxus&RPG=40&VEH=0&AVL=2">Maxus</a><a href="listing.php?MOD=Maybach&RPG=40&VEH=0&AVL=2">Maybach</a><a href="listing.php?MOD=Mazda&RPG=40&VEH=0&AVL=2">Mazda</a><a href="listing.php?MOD=McLaren&RPG=40&VEH=0&AVL=2">McLaren</a><a href="listing.php?MOD=Mercedes-Benz&RPG=40&VEH=0&AVL=2">Mercedes-Benz</a>
I have tried the below code: 我尝试了以下代码:
r = requests.get('https://www.sgcarmart.com/used_cars/listing.php?RPG=40&MOD=audi&VEH=0&RGD=&AVL=2&ORD=')
soup = BeautifulSoup(r.text, 'html.parser')
res_2 = soup.find_all('div', attrs={'id': 'make_attach_menu_child'})
makes = re.findall(r'<a href=\"listing.php\?MOD\=.+?\>(.+)', str(res_2))
You can start by making a request to the url : https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2 您可以从以下网址开始请求: https : //www.sgcarmart.com/used_cars/listing.php? MOD = Foton & RPG =40& VEH =0& AVL =2
import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")
Then parse it by BeautifulSoup: 然后通过BeautifulSoup解析它:
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')
The make list is in the div with class: make_child dropdown_child So we find it : 生成列表位于div中,其类为:make_child dropdown_child因此,我们找到了它:
lis = soup.find('div',attrs={'class':'make_child dropdown_child'})
Now each make is in a tag so I looped through the lis and appended the a text to a list: 现在,每个品牌都放在一个标签中,因此我遍历了lis并将文本附加到列表中:
makes = []
for i in lis.findAll('a') :
makes.append (i.text)
Then printing the list gives me the following output: 然后打印列表,将得到以下输出:
Compiling the whole code : 编译整个代码:
import requests
r = requests.get ("https://www.sgcarmart.com/used_cars/listing.php?MOD=Foton&RPG=40&VEH=0&AVL=2")
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text,'html5lib')
lis = soup.find('div',attrs={'class':'make_child dropdown_child'})
makes = []
for i in lis.findAll('a') :
makes.append (i.text)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.