从下拉列表中选择后抓取网页
[英]scrape webpage after selecting from dropdown
问题描述
转到https://www.comparis.ch/carfinder/default 时，“Modell”下拉列表处于非活动状态，直到您从“Marke”下拉列表中选择某些内容。 我的目标是通过浏览所有“Marke”值的列表来抓取所有“Modell”值。 由于从“标记”下拉列表中选择某些内容时 URL 不会更改，因此我正在努力实现这一目标。 我可以使用此代码轻松获得“Marke”值列表：
from bs4 import BeautifulSoup
newest_secondhand_cars = 'https://www.comparis.ch/carfinder/default'
newest_secondhand_cars = requests.get(newest_secondhand_cars)
newest_secondhand_cars = newest_secondhand_cars.content
soup = BeautifulSoup(newest_secondhand_cars, "lxml")
marken = [str(el).split('>')[1].split('<')[0] for el in list(soup.find('select', {'id': 'ddlMake'}).find_all('option'))][1:]
print(marken)
我现在如何动态插入“Marke”值，然后抓取所有“Modell”值？ 我认为这个答案非常接近，但我没有设法完成它： 从下拉列表中的选定选项中抓取响应。 这将是尝试使用“Marke”值执行 POST 请求，然后从响应中抓取“Modell”值的代码（这不起作用）：
from bs4 import BeautifulSoup
import requests
url = 'https://www.comparis.ch/carfinder/default'
with requests.Session() as session:
session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}
# parsing parameters
response = session.get(url)
soup = BeautifulSoup(response.content, 'lxml')
data = {
'MakeList': 'BMW',
#'ctl00$ctl00$txtSearchWord': '',
#'__EVENTTARGET': soup.find('input', {'name': '__EVENTTARGET'}).get('value', ''),
#'__EVENTARGUMENT': soup.find('input', {'name': '__EVENTARGUMENT'}).get('value', ''),
#'__LASTFOCUS': soup.find('input', {'name': '__LASTFOCUS'}).get('value', ''),
#'__VIEWSTATE': soup.find('input', {'name': '__VIEWSTATE'}).get('value', ''),
#'__VIEWSTATEGENERATOR': soup.find('input', {'name': '__VIEWSTATEGENERATOR'}).get('value', ''),
#'__EVENTVALIDATION': soup.find('input', {'name': '__EVENTVALIDATION'}).get('value', ''),
}
# parsing data
response = session.post(url, data=data)
soup = BeautifulSoup(response.content, 'lxml')
print(soup.find('select', {'id': 'ddlModelGroup'}))
1 个解决方案
解决方案1
1 已采纳 2019-12-26 12:56:49
import requests
from bs4 import BeautifulSoup
from itertools import zip_longest
r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')
marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
for name in item.findAll("option")[1:]:
marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
for num in item.findAll("option")[1:]:
tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
print(f"{'*' * 10}Extracting: {a}{'*' * 10}\n")
r = requests.get(
f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
data = list(item['Value'].strip() for item in r)
print(f"{data}\n")
model.append(data)
输出样本：
**********Extracting: Aixam**********
['300', '325', '400', '500', '600', '721', '751', 'Crossline', 'MAC', 'Scouty']
等等....
或使用以下方法：
import requests
from bs4 import BeautifulSoup
from itertools import zip_longest
r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')
marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
for name in item.findAll("option")[1:]:
marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
for num in item.findAll("option")[1:]:
tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
print(f"Extracting: {a}")
r = requests.get(
f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
data = list(item['Value'] for item in r)
model.append(data)
for a, b in zip_longest(marke, model):
print("Marke: {:<20}, Model: {:<100}".format(
a, ", ".join(str(i).strip() for i in b)))
Marke: Aixam , Model: 300, 325, 400, 500, 600, 721, 751, Crossline, MAC,Scouty
Marke: Alfa Romeo , Model: Stelvio, 145, 146, 147, 155, 156, 159, 164, 166, 1750, 33, 4C, 6, 75, 8C, 90, Alfasud, Alfetta, Arna, Brera, Giulia, Giulietta, GT, GTV, Mito, Montreal, Spider, Zagato
Marke: American Motors , Model: Eagle
Marke: Artega , Model: GT
Marke: Aston Martin , Model: Cygnet, DB 11, DB 7, DB 9, DBS, Lagonda, Rapide, V12 Vanquish, V12 Vantage, V8, V8 Vantage, Virage
Marke: Audi , Model: 100, 200, 80, 90, A1, A2, A3, A4, A4 Allroad, A5, A6, A6 Allroad, A7, A8, Cabriolet, Coupé, e-Tron, Q2, Q3, Q5, Q7, Q8, Quattro, R8, RS Q3, RS3, RS4, RS5, RS6, S1, S3, S4, S5, S6, S7, S8, SQ5, TT, V8
Marke: Austin , Model: Healey, Maestro, Metro, Montego
Marke: Bentley , Model: Arnage, Azure, Bentayga, Brooklands, Continental, Flying Spur, Mulsanne, Turbo
Marke: Bertone , Model: Freeclimber
Marke: BMW , Model: 1 Series, 2 Series, 2002, 3 Series, 4 Series, 5 Series, 6 Series, 7 Series, 8 Series, i3, i8, M-Series, X-Series, Z-Series
Marke: Bmw-Alpina , Model: B10, B12, B3, B4, B5, B6, B7, B8, D10, D3, D4, D5,Roadster, XD3, XD4, Z8
Marke: Bugatti , Model: EB, Veyron
Marke: Buick , Model: Century, Electra, Park Avenue, Regal, Riviera, Skylark
Marke: Cadillac , Model: Allanté, ATS, BLS, CT, CTS, Deville, Eldorado, Escalade, Fleetwood, Seville, SRX, STS, XLR, XT
Marke: Caterham , Model: Seven, Super Seven
Marke: Chevrolet , Model: Alero, Astro, Aveo, Beretta, Blazer, Camaro, Caprice, Captiva, Cavalier, Celebrity, Citation, Corsica, Corvair, Corvette, Cruze, Epica, Evanda, HHR, Impala, Kalos, Lacetti, Malibu, Matiz, Nubira, Orlando, Spark, Tacuma, Tahoe, TrailBlazer, Trans-Sport, Trax, Volt
Marke: Chrysler , Model: 300C, 300M, Crossfire, Daytona, ES, Grand Voyager,GS, GTS, Le Baron, Neon, New Yorker, PT Cruiser, Saratoga, Sebring, Stratus, Valiant, Viper, Vision, Voyager
Marke: Citroen , Model: 2CV, Acadiane, AX, Berlingo, Berlingo Electric, BX, C 15, C 25, C 35, C1, C2, C3, C3 Picasso, C3 Pluriel, C4, C4 Aircross, C4 Cactus, C4 Grand Picasso, C4 Picasso, C5, C6, C8, C-Crosser, CX, C-Zero, DS3, DS4, DS5, E-Mehari, Evasion, GSA, Jumper, Jumpy, LNA, Nemo, Saxo, Spacetourer, Visa, Xantia, XM, Xsara, ZX
Marke: Cupra , Model: Ateca
Marke: Dacia , Model: Dokker, Duster, Lodgy, Logan, Sandero
Marke: Daewoo , Model: Espero, Evanda, Kalos, Korando, Lacetti, Lanos, Leganza, Matiz, Musso, Nexia, Nubira, Rexton, Tacuma
Marke: Daihatsu , Model: Applause, CAB, Charade, Charmant, Copen, Cuore, Feroza, Gran Move, Hijet, Materia, Move, Pick-up, Rocky, Sirion, Taft, Terios, Trevis, YRV
Marke: Daimler , Model: X300, X308, XJ 40, XJ Serie III, XJ Serie V
Marke: De Tomaso , Model: Guarà, Pantera
Marke: DFM , Model: Mini
Marke: Dfsk , Model: C-Serie, K-Serie, V-Serie
Marke: Dodge , Model: Avenger, Caliber, Challenger, Dart, Durango, Journey, Magnum, Nitro, Ram, Viper
Marke: Donkervoort , Model: D8, S8
Marke: Ds Automobiles , Model: DS3, DS4, DS5, DS7
Marke: Ferrari , Model: 308, 328, 348, 412, 456, 512, 575, 599, 612, 812, California, Dino, Enzo, F12, F355, F360, F40, F430, F458, F488, F50, F550, FF, GTC, Laferrari, Mondial, Portofino, Testarossa
Marke: Fiat , Model: 124, 126, 127, 130, 131, 500, 500L, 500X, 595, 850, Argenta, Barchetta, Bertone, Brava, Bravo, Cinquecento, Coupé, Croma, Dino, Doblo, Ducato, Fiorino, Freemont, Fullback, Idea, Marea, Multipla, Palio, Panda, Punto, Qubo, Regata, Ritmo, Scudo, Sedici, Seicento, Spider, Stilo, Strada, Talento, Tempra, Tipo, Topolino, Ulysse, Uno
Marke: Fisker , Model: Karma
Marke: Ford , Model: B-Max, Capri, C-Max, Cortina, Courier, Ecosport, Edge, Escort, Fiesta, Focus, Fusion, Galaxy, Granada, Grand C-MAX, KA, Kuga, Maverick, Mercury, Mondeo, Mustang, Orion, Puma, Ranger, Scorpio, Sierra, S-Max, Streetka, Taunus,Tourneo, Tourneo Connect, Tourneo Courier, Tourneo Custom, Transit, Transit Connect, Transit Courier, Transit Custom
Marke: Ford (USA) , Model: Aerostar, Cougar, Explorer, GT, Mustang, Probe, Taurus, Thunderbird, Windstar
Marke: Giotti Victoria , Model: Gladiator
Marke: Honda , Model: Accord, Acty, Ballade, Civic, Concerto, CR-V, CR-X, CR-Z, FR-V, HR-V, Insight, Integra, Jazz, Joker, Legend, Logo, NSX, Prelude, Quintet,S2000, Shuttle, Stream
Marke: HS , Model: Cobra
Marke: Hummer , Model: H2, H3
Marke: Hyundai , Model: Accent, Atos, Coupé, Elantra, Galloper, Genesis, Getz, Grandeur, H 100, H-1, H350, i10, i20, i30, i40, Ioniq, iX20, iX35, iX55, Kona, Lantra, Matrix, Nexo, Pony, Santa Fe, Santamo, Scoupe, Sonata, Starex, Terracan, Trajet, Tucson, Veloster, XG
Marke: Infiniti , Model: EX, FX, G, M, Q, QX
Marke: Innocenti , Model: Mini
Marke: Isuzu , Model: Campo, D-Max, KB, L, Midi, N, NKR, NLR, NMR, NNR, NPR, Pick Up, WFR, WFS
Marke: Isuzu (GM) , Model: Trooper
Marke: Isuzu (J) , Model: Gemini, Piazza
Marke: Iveco , Model: 50, Daily, Eco-daily, Massif, Turbo-daily
Marke: Jaguar , Model: Concept, E-Pace, E-Type, F-Pace, F-Type, I-Pace, MK, S-Type, XE, XF, XJ, XJR, XJ-S, XK, X-Type
Marke: Jeep , Model: Cherokee, CJ-7, CJ-8, Commander, Compass, Grand Cherokee, Kaiser, Patriot, Renegade, Wagoneer, Willys, Wrangler
Marke: KIA , Model: Carens, Carnival, Cee'd, Cerato, Clarus, K2500, K2900, Leo, Magentis, Niro, Opirus, Optima, Picanto, Pregio, Pride, Rio, Sephia, Shuma, Sorento, Soul, Sportage, Stinger, Stonic, Venga
Marke: KTM , Model: X-Bow
Marke: Lada , Model: 110, 111, 112, 1500, Kalina, Niva, Nova, Priora, Samara
Marke: Lamborghini , Model: Aventador, Countach, Diablo, Gallardo, Huracan, Jalpa, LM, Murciélago, Urus
Marke: Lancia , Model: A 112, Beta, Dedra, Delta, Flaminia, Flavia, Fulvia, Gamma, HPE, Kappa, Lybra, Musa, Phedra, Prisma, Thema, Thesis, Trevi, Voyager, Y, Ypsilon, Zeta
Marke: Land Rover , Model: 109, 110, 88, 90, Defender, Discovery, Evoque, Freelander, Range Rover, Range Rover Sport, Range Rover Velar
Marke: Lexus , Model: CT, GS, IS, LC, LS, NX, RC, RX, SC, UX
Marke: Ligier , Model: Be Up, Nova
Marke: Lotus , Model: 3-Eleven, Elan, Elise, Esprit, Europa, Evora, Exige, Super Seven
Marke: MAN , Model: TGE
Marke: Maserati , Model: 3200, Biturbo, Coupé, Ghibli, Grancabrio, Granturismo, Karif, Levante, Quattroporte, Shamal, Spyder
Marke: Mazda , Model: 121, 2, 3, 323, 5, 6, 626, 929, B 2500, BT-50, CX-3, CX-30, CX-5, CX-7, CX-9, Demio, E, MPV, MX-3, MX-5, MX-6, Premacy, RX-7, RX-8, Tribute, Xedos
Marke: Mclaren , Model: 540c, 570gt, 570s, 600lt, 650S, 675lt, 720S, MP4
Marke: Mega , Model: Mega
Marke: Mercedes-Benz , Model: 100, 190, 207, 208, 209, 210, 220, 230, 250, 280, 307, 308, 309, 310, 350, 400, 407, 409, 410, 450, 508, 510, 609, 611, 612, A-Klasse, B-Klasse, Citan, C-Klasse, CLA-Klasse, CLC-Klasse, CLK-Klasse, CL-Klasse, CLS-Klasse, E-Klasse, EQC, G-Klasse, GLA-Klasse, GLC-Klasse, GLE-Klasse, GLK-Klasse, GL-Klasse, GLS-Klasse, GT, Marco Polo, M-Klasse, R-Klasse, SEC-Klasse, S-Klasse, SLC-Klasse, SLK-Klasse,SL-Klasse, SLS-Klasse, Sprinter, Vaneo, Viano, Vito, V-Klasse, X-Klasse
Marke: MG , Model: F, Maestro, Metro, MGB, Montego, TF, ZR, ZS, ZT
Marke: Minelli , Model: TF
Marke: Mini , Model: Cabriolet, Clubman, Clubvan, Countryman, Coupé, Mini, Paceman, Roadster
Marke: Mitsubishi , Model: 3000 GT, ASX, Attrage, Canter, Carisma, Colt, Cordia, Eclipse, Galant, Grandis, i Miev, L 200, L 300, L 400, Lancer, Outlander, Pajero, Sapporo, Sigma, Space Gear, Space Runner, Space Star, Space Wagon, Starion, Tredia
Marke: Mitsubishi Fuso , Model: Canter
Marke: Morgan , Model: 4/4, Aero 8, Plus
Marke: Nissan , Model: Almera, Atleon, Bluebird, Cabstar, Cabstar E, Cherry, Cube, Double, e-NV200, GT-R, Interstar, Juke, King-/Double, Kubistar, Laurel, Leaf,Maxima, Micra, Murano, Navara, Note, Np300, NT400, Nv200, Nv300, Nv400, NX, Pathfinder, Patrol, Pixo, Prairie, Primastar, Primera, Pulsar, Qashqai, Serena, Silvia, Stanza, Sunny, SX, Terrano, Tiida, Trade, Urvan, Vanette, X-Trail, Z, ZX
Marke: Oldsmobile , Model: Cutlass, Delta 88, Ninety Eight, Toronado
Marke: Opel , Model: Adam, Admiral, Agila, Ampera, Antara, Ascona, Astra, Blitz, Calibra, Campo, Cascada, Combo, Commodore, Corsa, Crossland, Frontera, Grandland, GT, Insignia, Kadett, Karl, Manta, Meriva, Mokka, Monterey, Monza, Movano, Olympia, Omega, Rekord, Senator, Signum, Sintra, Speedster, Tigra, Vectra, Vivaro, Zafira
Marke: Peugeot , Model: 1007, 104, 106, 107, 108, 2008, 205, 206, 207, 208, 3008, 304, 305, 306, 307, 308, 309, 4007, 4008, 404, 405, 406, 407, 5008, 504, 505, 508, 604, 605, 607, 806, 807, Bipper, Boxer, Expert, iON, J 5, J 9, Partner, RCZ, Rifter, Traveler
Marke: Piaggio , Model: Maxxi, Porter, Quargo
Marke: Pontiac , Model: 6000, Bonneville, Firebird, Grand-AM, Grand-Prix, Trans Am, Trans Sport
Marke: Porsche , Model: 356, 911, 924, 928, 944, 968, Boxster, Carrera GT,Cayenne, Cayman, Macan, Panamera
Marke: Puch , Model: G, Haflinger, Pinzgauer
Marke: Qvale , Model: Mangusta
Marke: Reliant , Model: Scimitar
Marke: Renault , Model: 4 F, Alaskan, Alpine, Avantime, Captur, Clio, Espace, Express, Fluence, Fuego, Grand Espace, Grand Scénic, Kadjar, Kangoo, Koleos, Laguna, Latitude, Master, Mégane, Modus, R11, R12, R14, R16, R18, R19, R20, R21, R25, R30, R4, R5, R9, Safrane, Scénic, Spider, Super 5, Talisman, Trafic, Twingo, Twizy, Vel Satis, Wind, Zoe
Marke: Renault Trucks , Model: B 120, B 90, Mascott, Master, Maxity
Marke: Rolls-Royce , Model: Corniche, Cullinan, Dawn, Flying Spur, Ghost, Park Ward, Phantom, Silver Dawn, Silver Seraph, Silver Spirit, Silver Spur, Touring, Wraith
Marke: Rover , Model: 200, 25, 2600, 3500, 400, 45, 600, 75, 800, Mini, Streetwise
Marke: Saab , Model: 90, 900, 9000, 9-3, 9-4X, 9-5, 99
Marke: Seat , Model: Alhambra, Altea, Altea XL, Arona, Arosa, Ateca, Cordoba, Exeo, Ibiza, Inca, Leon, Malaga, Marbella, MII, Ronda, Tarraco, Terra, Toledo
Marke: Secma , Model: F16
Marke: Skoda , Model: 135, 136, Citigo, Fabia, Favorit, Felicia, Forman,Kamiq, Karoq, Kodiaq, Octavia, Praktik, Rapid, Roomster, Scala, Superb, Yeti
Marke: Smart , Model: Forfour, Fortwo, Roadster
Marke: Sokon , Model: Mini Truck
Marke: Ssang Yong , Model: Actyon, Korando, Kyron, Musso, Rexton, Rodius, Tivoli, XLV
Marke: Subaru , Model: 1600, 1800, 700, BRZ, E10, E12, Forester, G3X Justy, Impreza, Justy, Legacy, Levorg, Outback, SVX, Trezia, Tribeca, Vivio, WRX, XT, XV
Marke: Suzuki , Model: Alto, Baleno, Carry, Celerio, Grand Vitara, Ignis, Jimny, Kizashi, Liana, LJ, SA, Samurai, SJ, Splash, Swift, SX4, SX4 S-Cross, Vitara, Wagon R, X-90
Marke: Talbot , Model: Horizon, Matra, Samba, Solara, Tagora
Marke: Tata , Model: Indica, Pick Up, Safari
Marke: Tazzari , Model: Zero
Marke: Tesla , Model: MODEL 3, Model S, Model X, Roadster
Marke: Think , Model: City
Marke: Toyota , Model: 4-Runner, Auris, Auris Touring Sports, Avensis, Avensis Verso, Aygo, Camry, Carina, Celica, C-HR, Corolla, Corolla Verso, Cressida, Crown, Dyna, GT, Hi-ace, Hi-lux, iQ, Land Cruiser, Lite-ace, Mirai, Model - F, MR 2, Paseo, Picnic, Previa, Prius, Prius+, Proace, Proace Verso, RAV-4, Sienna, Starlet, Supra, Tercel, Urban Cruiser, Verso, Verso-S, Yaris, Yaris Verso
Marke: Triumph , Model: Spitfire, TR 6
Marke: TVR , Model: Cerbera, Chimaera, Griffith, S4, T350, Tuscan, V8
Marke: Venturi , Model: Cabriolet, Coupé
Marke: Volvo , Model: 144, 164, 240, 340, 360, 440, 460, 480, 740, 760, 780, 850, 940, 960, C30, C70, PV, S40, S60, S60 CC, S70, S80, S90, V40, V40 CC, V50, V60, V60 CC, V70, V90, V90 CC, XC40, XC60, XC70, XC90
Marke: VW , Model: Amarok, Arteon, Beetle, Bora, Buggy, Caddy, CC, Corrado, Crafter, Cross Caddy, Crossgolf, Crosspolo, Crosstouran, Crossup, Eos, Fox, Golf, Golf Plus, Golf Sportsvan, Grand California, Jetta, Käfer, Karmann, LT - Serie, Lupo,Passat, Phaeton, Polo, Santana, Scirocco, Sharan, T1, T2, T4, T5, T6, Taro - Serie, T-Cross, Tiguan, Touareg, Touran, T-ROC, Typ 2 - Serie, Up, Vento
Marke: Wiesmann , Model: Coupé, Roadster
Marke: Zagato , Model: Hyena
刷新后如何从网页中抓取数据
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.