Web 用Python BS刮

Question

Trying to scrape some weather data off of Weather Underground. I haven't had any difficulty getting the data of interest until I came to getting the day/date, hi/lo temps, and forecast (ie. "Partly Cloudy"). Each is in a div without a class. The parent, of each, is a div with a class="obs-date" (see image below)

[WxUn HTML image][1]

Attempted code below with other options commented out. Each returns an empty list.

def get_wx(city, state):
    city=city.lower()
    state=state.lower()
    
    # get current conditions; 'weather' in url
    current_dict = get_current(city, state)

    # get forecast; 'forecast' in url
    f_url = f'https://www.wunderground.com/forecast/us/{state}/{city}'
    f_response = req.get(f_url)
    f_soup = BeautifulSoup(f_response.text, 'html.parser')
    cast_dates = f_soup.find_all('div', class_="obs-date")
    # cast_dates = f_soup.find_all('div', attrs={"class":"obs-date"})
    # cast_dates = f_soup.select('div.obs-date')
    print(cast_dates)
    
get_wx("Portland", "ME")

Any help with what I'm missing is appreciated.

Answer 1

As far as I can see the whole block you're trying to parse is driven by javascript, that's why you're getting empty results using beautifulsoup

The ADDITIONAL CONDITIONS part could be parsed completely using bs4 as well as everything below. Table at the end could be parsed using pandas .

To scrape javascript content, you can use requests-html or selenium libraries.

from requests_html import HTMLSession
import json

session = HTMLSession()
url = "https://www.wunderground.com/weather/us/me/portland"
response = session.get(url)
response.html.render(sleep=1)

data = []

current_date = response.html.find('.timestamp strong', first = True).text
weather_conditions = response.html.find('.condition-icon p', first = True).text
gusts = response.html.find('.medium-uncentered span', first = True).text
current_temp = response.html.find('.current-temp .is-degree-visible', first = True).text

data.append({
    "Last update": current_date,
    "Current weather": weather_conditions,
    "Temperature": current_temp,
    "Gusts": gusts,
})

print(json.dumps(data, indent = 2, ensure_ascii = False))

Output:

[
  {
    "Last update": "1:27 PM EDT on April 14, 2021",
    "Current weather": "Fair",
    "Temperature": "49 F",
    "Gusts": "13 mph"
  }
]