Selenium Python | 'find_elements_by_class_name' 不返回任何内容

Question

I am trying to scrape job titles from a dynamic job listing. When I use the function find_elements_by_class_name, the function doesn't return anything. I'm new to selenium so i'm not sure if i'm simply doing something incorrectly or misunderstanding the functionality.

The page im trying to scrape is: https://recruit.hirebridge.com/v3/CareerCenter/v2/?cid=7724

from selenium import webdriver
import time

#define the path for the chrome webdriver
chrome_path = r"C:/web/jobListing/chromedriver.exe"

#create a instance of the webdriver
driver = webdriver.Chrome(chrome_path)

driver.get("https://recruit.hirebridge.com/v3/CareerCenter/v2/?cid=7724")
time.sleep(10)
jobs = driver.find_elements_by_class_name("col-md-8 jobtitle")

print("starting print")
for job in jobs:
    print(job.text)

Answer 1

Seems like a bug? This works:

jobs = driver.execute_script("""
  return document.getElementsByClassName("col-md-8 jobtitle")
""")

Answer 2

Try:

jobs = driver.find_elements_by_xpath("//div[@class='col-md-8 jobtitle']/a")

I've switched the find element by class for xpath, this way you have more flexibility and it generally works better, I suggest you look into it!

Answer 3

Root Cause:

col-md-8 and jobtitle are 2 different classes. When you use find_element_by_class_name it will internally convert the class name to a css selector and try to find the element.

Below is the evidence where find_element_by_class_name uses the css internally.

Solution:

As Selenium internally uses the css you have to make sure the classes are clubbed together meaning class1.class2.class3 . In simple terms replace all white spaces with single dot in the class name from UI.

How to implement this to your situation:

You have to use the below syntax.

driver.find_element_by_class_name('col-md-8.jobtitle')