BeautifulSoup can't find HTML element by class

Question

This is the website I'm trying to scrape with Python:

https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8&_sacat=0&LH_Sold=1&LH_Complete=1&rt=nc&LH_ItemCondition=3000

I want to access the 'ul' element with the class of 'srp-results srp-list clearfix'. This is what I tried with requests and BeautifulSoup:

from bs4 import BeautifulSoup
import requests

url = 'https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8&_sacat=0&LH_Sold=1&LH_Complete=1&rt=nc&LH_ItemCondition=3000'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')

uls = soup.find_all('ul', attrs = {'class': 'srp-results srp-list clearfix'})

And the output is always an empty string. I also tried scraping the website with Selenium Webdriver and I got the same result.

Answer 1

First I was a little bit confused about your error but after a bit of debugging I figured out that: eBay dynamically generates that ul with JavaScript

So since you can't execute JavaScript with BeautifulSoup you have to use selenium and wait until the JavaScript loads that ul

Answer 2

It is probably because the content you are looking for is rendered by JavaScript After the page loads on a web browser this means that the web browser load that content after running javascript which you cannot get with requests.get request from python.

I would suggest to learn Selenium to Scrape the data you want