Web Scraping with Beautiful Soup in Python - JavaScript Table

Question

Im trying to scrape a table from a website but I cant seem to figure it out with Beautifulsoup in Python. Im not sure if its because of the table format, but I basically want to turn this table into a CSV.

from bs4 import BeautifulSoup
import requests

page = requests.geenter code heret("https://spotwx.com/products/grib_index.php?model=hrrr_wrfprsf&lat=41.03399&lon=-73.76291&tz=America/New_York&display=table")
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.prettify)

Any advice on how to isolate this data table? I've checked so many Beautifulsoup tutorials, but the HTML looks different than most references. Many thanks in advance for your help -

Answer 1

Try this. The table from that site generates dynamically so you can't get results using requests only.

from selenium import webdriver
from bs4 import BeautifulSoup
import csv

link = "https://spotwx.com/products/grib_index.php?model=hrrr_wrfprsf&lat=41.03399&lon=-73.76291&tz=America/New_York&display=table"

with open("spotwx.csv", "w", newline='') as infile:
    writer = csv.writer(infile)
    writer.writerow(['DateTime','Tmp','Dpt','Rh','Wh','Wd','Wg','Apcp','Slp'])
    with webdriver.Chrome() as driver:
        driver.get(link)
        soup = BeautifulSoup(driver.page_source, 'lxml')
        for item in soup.select("table#example tbody tr"):
            data = [elem.text for elem in item.select('td')]
            print(data)
            writer.writerow(data)