python：无法使用 BeautifulSoup 从 html 获取特定数据

Question

I am trying to use beautifulsoup and urllib to pull the given percentage from a particular webpage: https://app.safespace.io/api/display/live-occupancy/86fb9e11?view=percent . I am very new to stuff like this. Here is my spaghetti code:

import urllib.request

contentSource = urllib.request.urlopen('https://app.safespace.io/api/display/live-occupancy/86fb9e11?view=percent')
read_content = contentSource.read()

from bs4 import BeautifulSoup
soup = BeautifulSoup(read_content, 'html.parser')

try1 = soup.find("span", {"id": "occupancyPct"})

print(try1)

On the original webpage, when "inspect element"ing the percentage, the percentage actually shows up in the html, as highlighted .

However, my code's printed output is <span class="text-xl" id="occupancyPct" style="margin-bottom: auto;"></span>

Note how my code's output does NOT show the percentage in the output, unlike the actual page's html. What am I doing wrong?

I will also accept "You are stupid because X, and you should do Y instead", or some variation of that.

Answer 1

The problem is that the percentage isn't a static field, it's generated/calculated with JavaScript. As far as I know with this type of webscraping you only extract the sourcecode before a JavaScript or something was executed. Therefore this field keeps blank.. Instead of the chrome inspecting tool try the view of the raw source code, the field is unfortunatelly empty.

Here the JavaScript Code that fills the percentage field:

var setters = {
          bgClass: (function() {
            var bgSetter = getNumericClassSetter(elms.bg);

            return function(occupants) {
              // get percent, floored to the nearest 5 percent
              var pct = Number(`${ Number(`${occupants}e+2`) / Number(`${maxCapacity}e+2`) }e+2`);
              var floor = pct - (pct % 5);

              if (floor >= 100) {
                bgSetter(105);
              }
              else {
                bgSetter(floor);
              }
            };
          })(),
          occupancyPct: (function(occupants) {
            elms.occupancyPct.innerText = Math.min(100, Math.floor((occupants / maxCapacity) * 100)) + '%';
          }),
        };

As far as I can see the percentage is calculated with given variables. Could it be a soultion to calculate the percentage by your own code instead?

Answer 2

It is not showing percentages because that percentage is calculated later through javascript. And the HTML which you are getting is the initial one that does not have the percentage.

Answer 3

The answer is simple: You must use selenium

why?

You need a navigator, so the javascript code will be executed and the percentage that you are looking for will be there, in the code source of the page, all you have to do is to find a trick to get it.

Answer 4

The page is loaded dynamically, therefore requests won't support it. We can Selenium as an alternative to scrape the page.

Install it with: pip install selenium .

Download the correct ChromeDriver from here .

from time import sleep
from selenium import webdriver
from bs4 import BeautifulSoup

URL = "https://app.safespace.io/api/display/live-occupancy/86fb9e11?view=percent"
driver = webdriver.Chrome(r"c:\path\to\chromedriver.exe")
driver.get(URL)
# Wait for page to fully render
sleep(5)

soup = BeautifulSoup(driver.page_source, "html.parser")
print(soup.find("span", {"id": "occupancyPct"}).text)

driver.quit()