如何使用 class 在 div 之间查找文本

Question

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url="https://store.steampowered.com/search/?specials=1"
uClient=uReq(my_url)
page_html=uClient.read()
uClient.close()

page_soup=soup(page_html,"html.parser")
game_name_containers=page_soup.findAll('div',{"class":"col search_name ellipsis"})
print(len(game_name_containers))

for game_name_container in game_name_containers:
    game_name=game_name_container.span.text
    print(game_name)

It prints out 50 games but there are obviously more than 50. How do I print all of the games?

Answer 1

Your code is correctly printing the names and the count of the games returned in the HTML. If you open up the source code of the Steam URL, you'll see that they only initially give you 50 games' worth of data. You have to scroll down to be given more.

As mentioned in this answer , BeautifulSoup is not aware of the Javascript infinite scrolling that is occurring. You will either need a different tool or find a way to trigger Steam to return more data to you.

As mentioned in that answer, Selenium is a tool you could try.