scrape webpage after selecting from dropdown

Question

When going to https://www.comparis.ch/carfinder/default the "Modell" dropdown is inactive until you select something from the "Marke" dropdown. My goal is to scrape all "Modell" values by going through the list of all "Marke" values. As the URL does not change when selecting something from the "Marke" dropdown I am struggling to achieve this. A list of "Marke" values I can get easily with this code:

from bs4 import BeautifulSoup

newest_secondhand_cars = 'https://www.comparis.ch/carfinder/default'
newest_secondhand_cars = requests.get(newest_secondhand_cars)
newest_secondhand_cars = newest_secondhand_cars.content
soup = BeautifulSoup(newest_secondhand_cars, "lxml")

marken = [str(el).split('>')[1].split('<')[0] for el in list(soup.find('select', {'id': 'ddlMake'}).find_all('option'))][1:]

print(marken)

How can I now plug in the "Marke" values dynamically to then scrape alle the "Modell" values? This answer comes very close I think but I did not manage to get it done: scraping a response from a selected option in dropdown list . This would be the code trying to do a POST request with the "Marke" value and then scraping the "Modell" values from the response (which is not working):

from bs4 import BeautifulSoup
import requests

url = 'https://www.comparis.ch/carfinder/default'

with requests.Session() as session:
    session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}

    # parsing parameters
    response = session.get(url)
    soup = BeautifulSoup(response.content, 'lxml')

    data = {
        'MakeList': 'BMW',
        #'ctl00$ctl00$txtSearchWord': '',
        #'__EVENTTARGET': soup.find('input', {'name': '__EVENTTARGET'}).get('value', ''),
        #'__EVENTARGUMENT': soup.find('input', {'name': '__EVENTARGUMENT'}).get('value', ''),
        #'__LASTFOCUS': soup.find('input', {'name': '__LASTFOCUS'}).get('value', ''),
        #'__VIEWSTATE': soup.find('input', {'name': '__VIEWSTATE'}).get('value', ''),
        #'__VIEWSTATEGENERATOR': soup.find('input', {'name': '__VIEWSTATEGENERATOR'}).get('value', ''),
        #'__EVENTVALIDATION': soup.find('input', {'name': '__EVENTVALIDATION'}).get('value', ''),
    }

    # parsing data
    response = session.post(url, data=data)

    soup = BeautifulSoup(response.content, 'lxml')

    print(soup.find('select', {'id': 'ddlModelGroup'}))

Answer 1

import requests
from bs4 import BeautifulSoup
from itertools import zip_longest

r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')

marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
    for name in item.findAll("option")[1:]:
        marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
    for num in item.findAll("option")[1:]:
        tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
    print(f"{'*' * 10}Extracting: {a}{'*' * 10}\n")
    r = requests.get(
        f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
    data = list(item['Value'].strip() for item in r)
    print(f"{data}\n")
    model.append(data)

Output Sample:

**********Extracting: Aixam**********

['300', '325', '400', '500', '600', '721', '751', 'Crossline', 'MAC', 'Scouty']

And So On....

Or use the following:

import requests
from bs4 import BeautifulSoup
from itertools import zip_longest

r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')

marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
    for name in item.findAll("option")[1:]:
        marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
    for num in item.findAll("option")[1:]:
        tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
    print(f"Extracting: {a}")
    r = requests.get(
        f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
    data = list(item['Value'] for item in r)
    model.append(data)

for a, b in zip_longest(marke, model):
    print("Marke: {:<20}, Model: {:<100}".format(
        a, ", ".join(str(i).strip() for i in b)))

Marke: Aixam               , Model: 300, 325, 400, 500, 600, 721, 751, Crossline, MAC,Scouty
Marke: Alfa Romeo          , Model: Stelvio, 145, 146, 147, 155, 156, 159, 164, 166, 1750, 33, 4C, 6, 75, 8C, 90, Alfasud, Alfetta, Arna, Brera, Giulia, Giulietta, GT, GTV, Mito, Montreal, Spider, Zagato
Marke: American Motors     , Model: Eagle
Marke: Artega              , Model: GT
Marke: Aston Martin        , Model: Cygnet, DB 11, DB 7, DB 9, DBS, Lagonda, Rapide, V12 Vanquish, V12 Vantage, V8, V8 Vantage, Virage
Marke: Audi                , Model: 100, 200, 80, 90, A1, A2, A3, A4, A4 Allroad, A5, A6, A6 Allroad, A7, A8, Cabriolet, Coupé, e-Tron, Q2, Q3, Q5, Q7, Q8, Quattro, R8, RS Q3, RS3, RS4, RS5, RS6, S1, S3, S4, S5, S6, S7, S8, SQ5, TT, V8
Marke: Austin              , Model: Healey, Maestro, Metro, Montego
Marke: Bentley             , Model: Arnage, Azure, Bentayga, Brooklands, Continental, Flying Spur, Mulsanne, Turbo
Marke: Bertone             , Model: Freeclimber
Marke: BMW                 , Model: 1 Series, 2 Series, 2002, 3 Series, 4 Series, 5 Series, 6 Series, 7 Series, 8 Series, i3, i8, M-Series, X-Series, Z-Series
Marke: Bmw-Alpina          , Model: B10, B12, B3, B4, B5, B6, B7, B8, D10, D3, D4, D5,Roadster, XD3, XD4, Z8
Marke: Bugatti             , Model: EB, Veyron
Marke: Buick               , Model: Century, Electra, Park  Avenue, Regal, Riviera, Skylark
Marke: Cadillac            , Model: Allanté, ATS, BLS, CT, CTS, Deville, Eldorado, Escalade, Fleetwood, Seville, SRX, STS, XLR, XT
Marke: Caterham            , Model: Seven, Super Seven
Marke: Chevrolet           , Model: Alero, Astro, Aveo, Beretta, Blazer, Camaro, Caprice, Captiva, Cavalier, Celebrity, Citation, Corsica, Corvair, Corvette, Cruze, Epica, Evanda, HHR, Impala, Kalos, Lacetti, Malibu, Matiz, Nubira, Orlando, Spark, Tacuma, Tahoe, TrailBlazer, Trans-Sport, Trax, Volt
Marke: Chrysler            , Model: 300C, 300M, Crossfire, Daytona, ES, Grand Voyager,GS, GTS, Le  Baron, Neon, New  Yorker, PT Cruiser, Saratoga, Sebring, Stratus, Valiant, Viper, Vision, Voyager
Marke: Citroen             , Model: 2CV, Acadiane, AX, Berlingo, Berlingo Electric, BX, C 15, C 25, C 35, C1, C2, C3, C3 Picasso, C3 Pluriel, C4, C4 Aircross, C4 Cactus, C4 Grand Picasso, C4 Picasso, C5, C6, C8, C-Crosser, CX, C-Zero, DS3, DS4, DS5, E-Mehari, Evasion, GSA, Jumper, Jumpy, LNA, Nemo, Saxo, Spacetourer, Visa, Xantia, XM, Xsara, ZX
Marke: Cupra               , Model: Ateca
Marke: Dacia               , Model: Dokker, Duster, Lodgy, Logan, Sandero
Marke: Daewoo              , Model: Espero, Evanda, Kalos, Korando, Lacetti, Lanos, Leganza, Matiz, Musso, Nexia, Nubira, Rexton, Tacuma
Marke: Daihatsu            , Model: Applause, CAB, Charade, Charmant, Copen, Cuore, Feroza, Gran Move, Hijet, Materia, Move, Pick-up, Rocky, Sirion, Taft, Terios, Trevis, YRV
Marke: Daimler             , Model: X300, X308, XJ 40, XJ Serie III, XJ Serie V
Marke: De Tomaso           , Model: Guarà, Pantera
Marke: DFM                 , Model: Mini
Marke: Dfsk                , Model: C-Serie, K-Serie, V-Serie
Marke: Dodge               , Model: Avenger, Caliber, Challenger, Dart, Durango, Journey, Magnum, Nitro, Ram, Viper
Marke: Donkervoort         , Model: D8, S8
Marke: Ds Automobiles      , Model: DS3, DS4, DS5, DS7
Marke: Ferrari             , Model: 308, 328, 348, 412, 456, 512, 575, 599, 612, 812, California, Dino, Enzo, F12, F355, F360, F40, F430, F458, F488, F50, F550, FF, GTC, Laferrari, Mondial, Portofino, Testarossa
Marke: Fiat                , Model: 124, 126, 127, 130, 131, 500, 500L, 500X, 595, 850, Argenta, Barchetta, Bertone, Brava, Bravo, Cinquecento, Coupé, Croma, Dino, Doblo, Ducato, Fiorino, Freemont, Fullback, Idea, Marea, Multipla, Palio, Panda, Punto, Qubo, Regata, Ritmo, Scudo, Sedici, Seicento, Spider, Stilo, Strada, Talento, Tempra, Tipo, Topolino, Ulysse, Uno
Marke: Fisker              , Model: Karma
Marke: Ford                , Model: B-Max, Capri, C-Max, Cortina, Courier, Ecosport, Edge, Escort, Fiesta, Focus, Fusion, Galaxy, Granada, Grand C-MAX, KA, Kuga, Maverick, Mercury, Mondeo, Mustang, Orion, Puma, Ranger, Scorpio, Sierra, S-Max, Streetka, Taunus,Tourneo, Tourneo Connect, Tourneo Courier, Tourneo Custom, Transit, Transit Connect, Transit Courier, Transit Custom
Marke: Ford (USA)          , Model: Aerostar, Cougar, Explorer, GT, Mustang, Probe, Taurus, Thunderbird, Windstar
Marke: Giotti Victoria     , Model: Gladiator
Marke: Honda               , Model: Accord, Acty, Ballade, Civic, Concerto, CR-V, CR-X, CR-Z, FR-V, HR-V, Insight, Integra, Jazz, Joker, Legend, Logo, NSX, Prelude, Quintet,S2000, Shuttle, Stream
Marke: HS                  , Model: Cobra
Marke: Hummer              , Model: H2, H3
Marke: Hyundai             , Model: Accent, Atos, Coupé, Elantra, Galloper, Genesis, Getz, Grandeur, H 100, H-1, H350, i10, i20, i30, i40, Ioniq, iX20, iX35, iX55, Kona, Lantra, Matrix, Nexo, Pony, Santa  Fe, Santamo, Scoupe, Sonata, Starex, Terracan, Trajet, Tucson, Veloster, XG
Marke: Infiniti            , Model: EX, FX, G, M, Q, QX
Marke: Innocenti           , Model: Mini
Marke: Isuzu               , Model: Campo, D-Max, KB, L, Midi, N, NKR, NLR, NMR, NNR, NPR, Pick Up, WFR, WFS
Marke: Isuzu (GM)          , Model: Trooper
Marke: Isuzu (J)           , Model: Gemini, Piazza
Marke: Iveco               , Model: 50, Daily, Eco-daily, Massif, Turbo-daily
Marke: Jaguar              , Model: Concept, E-Pace, E-Type, F-Pace, F-Type, I-Pace, MK, S-Type, XE, XF, XJ, XJR, XJ-S, XK, X-Type
Marke: Jeep                , Model: Cherokee, CJ-7, CJ-8, Commander, Compass, Grand  Cherokee, Kaiser, Patriot, Renegade, Wagoneer, Willys, Wrangler
Marke: KIA                 , Model: Carens, Carnival, Cee'd, Cerato, Clarus, K2500, K2900, Leo, Magentis, Niro, Opirus, Optima, Picanto, Pregio, Pride, Rio, Sephia, Shuma, Sorento, Soul, Sportage, Stinger, Stonic, Venga
Marke: KTM                 , Model: X-Bow
Marke: Lada                , Model: 110, 111, 112, 1500, Kalina, Niva, Nova, Priora, Samara
Marke: Lamborghini         , Model: Aventador, Countach, Diablo, Gallardo, Huracan, Jalpa, LM, Murciélago, Urus
Marke: Lancia              , Model: A 112, Beta, Dedra, Delta, Flaminia, Flavia, Fulvia, Gamma, HPE, Kappa, Lybra, Musa, Phedra, Prisma, Thema, Thesis, Trevi, Voyager, Y, Ypsilon, Zeta
Marke: Land Rover          , Model: 109, 110, 88, 90, Defender, Discovery, Evoque, Freelander, Range  Rover, Range Rover Sport, Range Rover Velar
Marke: Lexus               , Model: CT, GS, IS, LC, LS, NX, RC, RX, SC, UX
Marke: Ligier              , Model: Be  Up, Nova
Marke: Lotus               , Model: 3-Eleven, Elan, Elise, Esprit, Europa, Evora, Exige, Super Seven
Marke: MAN                 , Model: TGE
Marke: Maserati            , Model: 3200, Biturbo, Coupé, Ghibli, Grancabrio, Granturismo, Karif, Levante, Quattroporte, Shamal, Spyder
Marke: Mazda               , Model: 121, 2, 3, 323, 5, 6, 626, 929, B 2500, BT-50, CX-3, CX-30, CX-5, CX-7, CX-9, Demio, E, MPV, MX-3, MX-5, MX-6, Premacy, RX-7, RX-8, Tribute, Xedos
Marke: Mclaren             , Model: 540c, 570gt, 570s, 600lt, 650S, 675lt, 720S, MP4
Marke: Mega                , Model: Mega
Marke: Mercedes-Benz       , Model: 100, 190, 207, 208, 209, 210, 220, 230, 250, 280, 307, 308, 309, 310, 350, 400, 407, 409, 410, 450, 508, 510, 609, 611, 612, A-Klasse, B-Klasse, Citan, C-Klasse, CLA-Klasse, CLC-Klasse, CLK-Klasse, CL-Klasse, CLS-Klasse, E-Klasse, EQC, G-Klasse, GLA-Klasse, GLC-Klasse, GLE-Klasse, GLK-Klasse, GL-Klasse, GLS-Klasse, GT, Marco Polo, M-Klasse, R-Klasse, SEC-Klasse, S-Klasse, SLC-Klasse, SLK-Klasse,SL-Klasse, SLS-Klasse, Sprinter, Vaneo, Viano, Vito, V-Klasse, X-Klasse
Marke: MG                  , Model: F, Maestro, Metro, MGB, Montego, TF, ZR, ZS, ZT
Marke: Minelli             , Model: TF
Marke: Mini                , Model: Cabriolet, Clubman, Clubvan, Countryman, Coupé, Mini, Paceman, Roadster
Marke: Mitsubishi          , Model: 3000 GT, ASX, Attrage, Canter, Carisma, Colt, Cordia, Eclipse, Galant, Grandis, i Miev, L 200, L 300, L 400, Lancer, Outlander, Pajero, Sapporo, Sigma, Space  Gear, Space  Runner, Space  Star, Space  Wagon, Starion, Tredia
Marke: Mitsubishi Fuso     , Model: Canter
Marke: Morgan              , Model: 4/4, Aero  8, Plus
Marke: Nissan              , Model: Almera, Atleon, Bluebird, Cabstar, Cabstar E, Cherry, Cube, Double, e-NV200, GT-R, Interstar, Juke, King-/Double, Kubistar, Laurel, Leaf,Maxima, Micra, Murano, Navara, Note, Np300, NT400, Nv200, Nv300, Nv400, NX, Pathfinder, Patrol, Pixo, Prairie, Primastar, Primera, Pulsar, Qashqai, Serena, Silvia, Stanza, Sunny, SX, Terrano, Tiida, Trade, Urvan, Vanette, X-Trail, Z, ZX
Marke: Oldsmobile          , Model: Cutlass, Delta  88, Ninety  Eight, Toronado
Marke: Opel                , Model: Adam, Admiral, Agila, Ampera, Antara, Ascona, Astra, Blitz, Calibra, Campo, Cascada, Combo, Commodore, Corsa, Crossland, Frontera, Grandland, GT, Insignia, Kadett, Karl, Manta, Meriva, Mokka, Monterey, Monza, Movano, Olympia, Omega, Rekord, Senator, Signum, Sintra, Speedster, Tigra, Vectra, Vivaro, Zafira
Marke: Peugeot             , Model: 1007, 104, 106, 107, 108, 2008, 205, 206, 207, 208, 3008, 304, 305, 306, 307, 308, 309, 4007, 4008, 404, 405, 406, 407, 5008, 504, 505, 508, 604, 605, 607, 806, 807, Bipper, Boxer, Expert, iON, J 5, J 9, Partner, RCZ, Rifter, Traveler
Marke: Piaggio             , Model: Maxxi, Porter, Quargo
Marke: Pontiac             , Model: 6000, Bonneville, Firebird, Grand-AM, Grand-Prix, Trans Am, Trans Sport
Marke: Porsche             , Model: 356, 911, 924, 928, 944, 968, Boxster, Carrera GT,Cayenne, Cayman, Macan, Panamera
Marke: Puch                , Model: G, Haflinger, Pinzgauer
Marke: Qvale               , Model: Mangusta
Marke: Reliant             , Model: Scimitar
Marke: Renault             , Model: 4 F, Alaskan, Alpine, Avantime, Captur, Clio, Espace, Express, Fluence, Fuego, Grand Espace, Grand Scénic, Kadjar, Kangoo, Koleos, Laguna, Latitude, Master, Mégane, Modus, R11, R12, R14, R16, R18, R19, R20, R21, R25, R30, R4, R5, R9, Safrane, Scénic, Spider, Super  5, Talisman, Trafic, Twingo, Twizy, Vel  Satis, Wind, Zoe
Marke: Renault Trucks      , Model: B 120, B 90, Mascott, Master, Maxity
Marke: Rolls-Royce         , Model: Corniche, Cullinan, Dawn, Flying  Spur, Ghost, Park  Ward, Phantom, Silver Dawn, Silver Seraph, Silver Spirit, Silver Spur, Touring, Wraith
Marke: Rover               , Model: 200, 25, 2600, 3500, 400, 45, 600, 75, 800, Mini, Streetwise
Marke: Saab                , Model: 90, 900, 9000, 9-3, 9-4X, 9-5, 99
Marke: Seat                , Model: Alhambra, Altea, Altea XL, Arona, Arosa, Ateca, Cordoba, Exeo, Ibiza, Inca, Leon, Malaga, Marbella, MII, Ronda, Tarraco, Terra, Toledo
Marke: Secma               , Model: F16
Marke: Skoda               , Model: 135, 136, Citigo, Fabia, Favorit, Felicia, Forman,Kamiq, Karoq, Kodiaq, Octavia, Praktik, Rapid, Roomster, Scala, Superb, Yeti
Marke: Smart               , Model: Forfour, Fortwo, Roadster
Marke: Sokon               , Model: Mini Truck
Marke: Ssang Yong          , Model: Actyon, Korando, Kyron, Musso, Rexton, Rodius, Tivoli, XLV
Marke: Subaru              , Model: 1600, 1800, 700, BRZ, E10, E12, Forester, G3X Justy, Impreza, Justy, Legacy, Levorg, Outback, SVX, Trezia, Tribeca, Vivio, WRX, XT, XV
Marke: Suzuki              , Model: Alto, Baleno, Carry, Celerio, Grand  Vitara, Ignis, Jimny, Kizashi, Liana, LJ, SA, Samurai, SJ, Splash, Swift, SX4, SX4 S-Cross, Vitara, Wagon  R, X-90
Marke: Talbot              , Model: Horizon, Matra, Samba, Solara, Tagora
Marke: Tata                , Model: Indica, Pick  Up, Safari
Marke: Tazzari             , Model: Zero
Marke: Tesla               , Model: MODEL 3, Model S, Model X, Roadster
Marke: Think               , Model: City
Marke: Toyota              , Model: 4-Runner, Auris, Auris Touring Sports, Avensis, Avensis Verso, Aygo, Camry, Carina, Celica, C-HR, Corolla, Corolla Verso, Cressida, Crown, Dyna, GT, Hi-ace, Hi-lux, iQ, Land Cruiser, Lite-ace, Mirai, Model  -  F, MR 2, Paseo, Picnic, Previa, Prius, Prius+, Proace, Proace Verso, RAV-4, Sienna, Starlet, Supra, Tercel, Urban  Cruiser, Verso, Verso-S, Yaris, Yaris Verso
Marke: Triumph             , Model: Spitfire, TR 6
Marke: TVR                 , Model: Cerbera, Chimaera, Griffith, S4, T350, Tuscan, V8
Marke: Venturi             , Model: Cabriolet, Coupé
Marke: Volvo               , Model: 144, 164, 240, 340, 360, 440, 460, 480, 740, 760, 780, 850, 940, 960, C30, C70, PV, S40, S60, S60 CC, S70, S80, S90, V40, V40 CC, V50, V60, V60 CC, V70, V90, V90 CC, XC40, XC60, XC70, XC90
Marke: VW                  , Model: Amarok, Arteon, Beetle, Bora, Buggy, Caddy, CC, Corrado, Crafter, Cross Caddy, Crossgolf, Crosspolo, Crosstouran, Crossup, Eos, Fox, Golf, Golf Plus, Golf Sportsvan, Grand California, Jetta, Käfer, Karmann, LT - Serie, Lupo,Passat, Phaeton, Polo, Santana, Scirocco, Sharan, T1, T2, T4, T5, T6, Taro  -  Serie, T-Cross, Tiguan, Touareg, Touran, T-ROC, Typ  2  -  Serie, Up, Vento
Marke: Wiesmann            , Model: Coupé, Roadster
Marke: Zagato              , Model: Hyena