尝试使用 BS4 从网站上使用 Python 抓取数据

Question

I am trying to import data from the URL(mentioned in the code). When I run the code, I do not get any information(like the plan name and rates) and gives me container div tags but not the contents. Also, I tried response.text but it gave me no results.I do not want to use Selenium. Is there a way to solve it?

from bs4 import BeautifulSoup
import urllib

from urllib.request import urlopen

URL="https://www.energymadeeasy.gov.au/plan?id=POW15475MBE3&postcode=2000"
response=urlopen(URL)
html_content=BeautifulSoup(response)
print(html_content)

or

soup=BeautifulSoup(requests.get(URL).text,'lxml')
print(soup)

I tried to extract the header using below

h1=html_content.find("div", {"class":"header-left"})
print(h1)

Answer 1

The website makes ajax call behind to load the data.

There are 2 xhr calls that are made to load the data. Probably you are looking at one of them.

import requests, json
res = requests.get("https://api.energymadeeasy.gov.au/plans/dpids/POW15475MBE3")
with open("data.json", "w") as f:
    json.dump(res.json(), f)

The saves the json to the file.

Sample data in file:

[{"planData": {"planType": "M", "tariffType": "TOU", "contract": [{"pricingModel": "TOU", "benefitPeriod": "1 year", "coolingOffDays": 10, "solarFit": [{"type": "R", "description": "Powerdirect Retailer Feed-in Tariff (exc. GST if any)", "rate": 9.5}], "additionalFeeInformation": "Additional fees and charges may apply. Please see the Powerdirect fee schedules at powerdirect.com.au/fees", "fee": [{"description": "Fee may be charged when reconnecting or reading your meter when you move into a property or change retailer. Includes GST. Fees may vary.", "amount": 12.55, "feeType": "ConnF", "feeTerm": "F"}, {"description": "Fee may be charged when reconnecting in other circumstances, such as after disconnection for non-payment. Includes GST. Fees may vary.", "amount": 12.55, "feeType": "RecoF", "feeTerm": "F"}, {"description": "Fee may be charged when disconnecting or reading your meter when you move out of a property or change retailer. Includes GST. Fees may vary.", "amount": 12.55, "feeType": "DiscoFMO", "feeTerm": "F"}, {"description": 
...
...
...