Python Expedia的网页抓取，如何找到正确的关键字

Question

I am learning about Python scraper. I take an simple exercise about find the cheap ticket in Expedia. Now, I meet some problems about how to find the right selector or accurate keyword. I use functions like select() and find(). I took too many tests about them but I still did it successfully. I always get empty list. How can I find the right selector or keywork in a better method? There is a part of my code. In it, I try to find the location of the input of Place: Flying from and the button Roundtrip.

    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'}
    url="https://www.expedia.com/Flights?langid=1033&semcid=US.MULTILOBF.GOOGLE.GT-c-EN.FLIGHT&semdtl=a1355852835.b125535175035.r1.g1kwd-12197061.i1.d1280328929841.e1c.j120181.k1.f11t1.n1.l1g.h1e.m1&gclid=CjwKCAiAws7uBRAkEiwAMlbZjjBMg2bBBbp59C6tXeHf-4sXVvc4ya7EflIKQGsaFgENRP_SbaNQrRoCsUoQAvD_BwE"  
    address_page1 = requests.get(url, headers=headers).content
    soup = BeautifulSoup(address_page1,'html.parser')

    find = soup.find_all(id='flight-origin-flp-airport_code')
    print(find)

    select1 = soup.select('#flight-origin-flp-airport_code')
    print(select1)

    select2 = soup.select('#gcw-flights-form-flp > div.cols-nested.ab25184-location > div > div > div.input-btn-group')
    print(select2)

Answer 1

Your approach is fundamentally flawed. Most of today's websites, including Expedia, are heavily JavaScript-based. The data you want may not even render on the page when you fetch it this way. You probably want to use a framework similar to Puppeteer which emulates the entire browser. A simple Python-based library will not be able to execute on-page JavaScript like your browser does. If you want to stick to Python, there may be a Puppeteer wrapper, but you'd have a much easier time just using Puppeteer and JS directly.

Answer 2

Searching a part of page you need is easier with using browser developer tools, for example Chrome Ctrl + Shift + C.

You can search manually in page source code in browser or in

print(BeautifulSoup(requests.get(url).text, 'html.parser').prettify())

Your python request probably got 'We can't tell if you're a human or a bot.' page instead of normal expedia page. You can try to reuse cookies from your browser with your requests session to get proper pages.

When you found node you needed visually you look for it's name and attributes, also for parent's name and attributes.

Roundtrip example

result = soup.find_all('label')
roundtrip = None
for label in result:
    if not 'id' in label.attrs:
        continue
    if 'roundtrip' in label.attrs['id']:
        roundtrip = label
        break
print(roundtrip)