使用beautifulsoup从页面中抓取特定的机场代码

Question

This is my first post, please feel free to let me know how I could be posting better and thanks in advance for the help.

I am learning how scrape data from webpages with python using BeautifulSoup, and am having difficulty scraping all the airports where loungebuddy operates.

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.loungebuddy.com/select/locations')
soup = BeautifulSoup(page.text, 'html.parser')
airport_code_html_lines = soup.find_all( attrs={'class': 'aiprt-code'})

This gets me very close, but I have extraneous data. The result I want is the second line in each of the results provided by:

    for airport_code in airport_code_html_lines:
        print(airport_code.prettify())

I'm trying to personalize this very simple case here:

https://medium.freecodecamp.org/how-to-scrape-websites-with-python-and-beautifulsoup-5946935d93fe

where the author pulls the price portion. However, when i try to do the equivalent of

price = price_box.text

I get this error:

AttributeError: ResultSet object has no attribute 'txt'. You're probably 
treating a list of items like a single item. Did you call find_all() when you 
meant to call find()?

Python guessed correctly, I'm using a find all...but I don't know how else to proceed.

I have tried using different print functions like

print(airport_code.strip('>'))

To see if I could strip or isolate the code by creating new variables or with creative print commands, but I get this:

TypeError: 'NoneType' object is not callable

I would love either direction in what to try next (considering changing the find_all to a find, and then creating a for loop....but that's intimidating to me. Hoping for a cleaner solution), or working code which will spit out my desired result. I hope to learn python both through this project and in the future, so any comments on my thought process are appreciated.

Thanks again

Answer 1

Simply replacing print(airport_code.prettify()) with print(airport_code.text) will give you the output you want.

Try the following code (made it a bit cleaner):

page = requests.get('https://www.loungebuddy.com/select/locations')
soup = BeautifulSoup(page.text, 'html.parser')

for country in soup.find_all('span', class_='aiprt-code'):
    print(country.text)

You can use soup.find_all('span', {'class': 'aiprt-code'}) instead of soup.find_all('span', class_='aiprt-code') too. It's the same thing.

Output:

BNE
SYD
BGI
BRU
...
...

Or if you want the countries in a list, you can use list comprehension as shown below. It helps in storing, using and modifying the data.

countries = [x.text for x in soup.find_all('span', class_='aiprt-code')]
print(countries)

Output:

['BNE', 'SYD', 'BGI', 'BRU', 'GIG', 'SOF', 'PNH', 'REP', ... ]

Answer 2

If you wanna have the airport code along with it's country name then you can try like below:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.loungebuddy.com/select/locations')
soup = BeautifulSoup(page.text, 'html.parser')
airport_code = {item.select_one("h2").text:item.select_one(".aiprt-code").text for item in soup.select(".country")}
print(airport_code)

Partial output:

{'India': 'BLR', 'Poland': 'KTW', 'Thailand': 'BKK', 'Croatia': 'ZAG',--so on--}