BeautifulSoup findAll returns empty list

Question

I want to write a code that collects the names of the games in a steam profile, but no matter what div I use in my code, an empty list is returned.

import requests
from bs4 import BeautifulSoup

r = requests.get('https://steamcommunity.com/profiles/76561198287287846/games/?tab=all')
soup = BeautifulSoup(r.content, 'lxml')

games = soup.findAll("div",{"class":"gameListRow"})

print(games)

Answer 1

As mentioned content is provided dynamically by JavaScript one approache could be use of selenium . In new code please use find_all() instead of old syntax findAll() .

Example

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service

url = f'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()

games = soup.find_all("div",{"class":"gameListRow"})

print(games)

Answer 2

A working example selenium with BeautifulSoup. Please just run the code.

import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup

url = 'https://steamcommunity.com/profiles/76561198287287846/games/?tab=all'

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
time.sleep(2)
driver.get(url)
time.sleep(5)


soup = BeautifulSoup(driver.page_source, 'lxml')
games = soup.findAll("div",{"class":"gameListRow"})
for game in games:
    name= game.find("div",class_="gameListRowItemName ellipsis").text
    print(name)

Output:

Dota 2
Counter-Strike: Global Offensive
Sea of Thieves
SCUM
Rocket League
Arma 3
ARK: Survival Evolved
DayZ
Rust
Garry's Mod
Unturned
Grand Theft Auto IV: The Complete Edition
Grand Theft Auto V
Euro Truck Simulator 2
Insurgency
PAYDAY 2
Mount & Blade: Warband
Valheim
War Thunder
Mount & Blade II: Bannerlord
RimWorld
Reign Of Kings
Robocraft
American Truck Simulator
theHunter: Call of the Wild™
Phasmophobia
Scrap Mechanic
Team Fortress 2
Cities: Skylines
Stellaris
UBOAT
Tiger Knight: Empire War
Warframe
Don't Starve Together
Holdfast: Nations At War
Fishing Planet
Fistful of Frags
Half-Life
Beat Cop
Of Guards And Thieves
Chivalry: Medieval Warfare
Terraria
911 Operator
We Need To Go Deeper
The Escapists 2
Business Tour - Online Multiplayer Board Game
Deceit
Crossout
Pummel Party
Paladins
Hand Simulator
Heroes & Generals
Ranch Simulator
Brawlhalla
Medal of Honor(TM) Multiplayer
Gang Beasts
Guns of Icarus Online
Tower Unite
Jalopy
College Kings
Stick Fight: The Game
We Were Here
Contraband Police: Prologue
Battlefield: Bad Company™ 2
Satellite Reign
Ylands
Prison Simulator: Prologue
Shadows of War
Battlerite
The Mean Greens - Plastic Warfare
War Trigger 2
Scribble It!
Radical Heights
Block N Load
Rubber Bandits: Summer Prologue
SinVR
Dead Maze
DC Universe Online
DiRT 3 Complete Edition
ATLAS
Paunch
Apex Legends
Crusader Kings II
Trophy Fishing 2
Amnesia: A Machine for Pigs
Amnesia: The Dark Descent
Argo
ARK: Survival Of The Fittest
Arma 3 Helicopters
Arma 3 Karts
Arma 3 Malden
Arma 3 Marksmen
Arma 3 Zeus
Armor Of Heroes
Artifact Foundry
Battlefield™ 2042 Open Beta
Brütal Legend
Cities: Skylines - Carols, Candles and Candy
Cities: Skylines - Match Day
College Bound
Company of Heroes 2
DCS World Steam Edition
Dungeons 2
Eador. Masters of the Broken World
F1 2015
Fallout Shelter
Friday the 13th: The Game
Galactic Civilizations II: Ultimate Edition
GROUND BRANCH
GROUND BRANCH CTE (Community Test Environment)
Guacamelee! Super Turbo Championship Edition
Guns of Icarus Alliance
Hacknet
Hacknet Official Soundtrack
Hide and Seek
HITMAN™
HITMAN™ 2
Human: Fall Flat
Kopanito All-Stars Soccer
Last Man Standing
Layers of Fear
Medal of Honor(TM) Single Player
Medal of Honor Pre-Order
Metro 2033
Mirage: Arcane Warfare
Mount & Blade: Warband - Napoleonic Wars
Of Guards and Thieves - Firefight
Of Guards and Thieves - Map Workshop
Outlast
Outlast: Whistleblower DLC
Paladins - Public Test
Penguins Arena: Sedna's World
Psychonauts
Quake Champions
RimWorld - Ideology
Rust - Staging Branch
Sanctum 2
Spacelords
Spec Ops: The Line
Survarium
Sven Co-op
The Darkness II
The Flame in the Flood
The Red Solstice
The Walking Dead
Tiger Knight
Totally Accurate Battlegrounds
Turbo Pug DX
Unreal Gold
Unturned - Early Access
World of Tanks Blitz