Problem with BS4 printing only certain parts of the web page

Question

I am having a problem with bs4 only finding some things in html. To be specific when I try to print span.nav2__menu-link-main-text it selects it and prints it without a problem but when I try to select other part of the page it probably selects it but It doesnt want to print it out. Here is the code that prints and the code that doesnt print:

Tried using different parsers other than lxml and none worked.

#This one prints

from bs4 import BeautifulSoup
import requests
import lxml

url = 'https://osu.ppy.sh/users/12008062'
res = requests.get(url)
soup = BeautifulSoup(res.text, 'lxml')

for i in soup.select('span.nav2__menu-link-main-text'):
    print(i.text)

#This one does not print

from bs4 import BeautifulSoup
import requests
import lxml

url = 'https://osu.ppy.sh/users/12008062'
res = requests.get(url)
soup = BeautifulSoup(res.text, 'lxml')

for i in soup.select('div.value-dispaly__value'):
    print(i.text)

I expect this program to print the current value of div.value-dispaly__value but when I start the program it prints nothing even tough I can see the value is 4000 when I inspect the page.

Answer 1

It seems that code you are willing to get is dynamically added to the web page by javascript. In order to update web js part, you have to use requests render() function.

Answer 2

Website page is javascript request rendering to get data, so you need to use automation library like selenium . download selenium web driver as per your browser requirement.

Download selenium web driver for chrome browser:

http://chromedriver.chromium.org/downloads

Install web driver for chrome browser:

https://christopher.su/2015/selenium-chromedriver-ubuntu/

Selenium tutorial:

https://selenium-python.readthedocs.io/

Replace your code to this:

from selenium import webdriver
from bs4 import BeautifulSoup
import time

driver = webdriver.Chrome('/usr/bin/chromedriver')
driver.get('https://osu.ppy.sh/users/12008062')
time.sleep(3)

soup = BeautifulSoup(driver.page_source, 'lxml')

for i in soup.find_all('div',{"class":"value-display__value"}):
    print(i.get_text())

O/P:

#47,514
#108
11d 19h 49m
44
4,000
11d 19h 49m
44
4,000
#47,514
#108
0
0