Beautiful Soup (Python) not seeing text inside of span

Question

I can't figure out why BS4 is not seeing the text inside of the span in the following scenario:

• Page: https://pypi.org/project/requests/
• Text I'm looking for - number of stars on the left hand side (around 43,000 at the time of writing)

My code:

stars = soup.find('span', {'class': 'github-repo-info__item', 'data-key': 'stargazers_count'}).text

also tried:

stars = soup.find('span', {'class': 'github-repo-info__item', 'data-key': 'stargazers_count'}).get_text()

Both return an empty string '' . The element itself seems to be located correctly (I can browse through parents / siblings in PyCharm debugger without a problem. Fetching text in other parts of the website also works perfectly fine. It's just the github-related stats that fail to fetch.

Any ideas?

Answer 1

Because this page use Javascript to load the page dynamically.So you couldn't get it directly by response.text

The source code of the page:

You could crawl the API directly:

import requests

r = requests.get('https://api.github.com/repos/psf/requests')
print(r.json()["stargazers_count"])

Result:

43010

Answer 2

Using bs4, we can't scrape stars rate.

After inspecting the site, please check response html. There, there is class information named "github-repo-info__item", but there is no text information.

in this case, use selenium .