Python 请求输出与预期输出不同

Question

I am trying to scrape the Generation table off the following site .

I have the following code:

import requests
from bs4 import BeautifulSoup

source = requests.get('http://reneweconomy.com.au/nem-watch/', headers={'User-Agent': 'Mozilla/5.0'})
soup = BeautifulSoup(source.text, 'html.parser')

table = soup.table["database table"].strip()
print(table)

However this code cannot find any table in the scraped page even though it is clearly there when looking through inspect. Could this be an issue with site not scraping correctly?

Thanks

Answer 1

This page uses JavaScript to load data and to create table.

Using DevTools in Firefox/Chrome I found it load it from

https://ausrealtimefueltype.global-roam.com/api/SeriesSnapshot?time=

as JSON data

---

import requests

headers = {'User-Agent': 'Mozilla/5.0'}

url = 'https://ausrealtimefueltype.global-roam.com/api/SeriesSnapshot?time='

r = requests.get(url,  headers=headers)
data = r.json()

for item in data['seriesCollection']:
    #for key, value in item.items():
    #    print(key, value)
    print('region:', item['metadata']['region']['name'])
    print('fuel type:', item['metadata']['fuelType']['name'])
    print('value:', item['value'])
    print('---')

Result

region: Queensland
fuel type: Black Coal
value: 5536.51307
---
region: Queensland
fuel type: Gas
value: 560.24621
---
region: Queensland
fuel type: Liquid Fuel
value: 0.0
---
region: Queensland
fuel type: Other
value: 23.9
---
region: Queensland
fuel type: Hydro
value: 18.415
---

# etc.