使用xpath selenium查找元素

Question

I'm following this tutorial in order to learn how to build a webscraper to get jobs listings. Now I'm trying to follow along with another website. I run into the problem that I don't know how to extract the links of the individual job listings.

When I'm inspecting the page I've found the element I need

When I copy the xpath and use it in my code I get an error. What am I doing wrong?

import selenium 
base_url = "https://www.nationalevacaturebank.nl"     
start_url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO" 
driver = webdriver.Firefox()    
elem = driver.find_element_by_xpath("//*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a")

>>NoSuchElementException: Message: Unable to locate element: //*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a

Answer 1

First, you should be connecting to your website...

Second, you should use waits you can read about it here

the code should look something like:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 


base_url = "https://www.nationalevacaturebank.nl"     
start_url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO" 
my_xpath = '//*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a'
driver = webdriver.Firefox()
driver.get(start_url)  
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, my_xpath)))

Edit

I order to get all the links of the individual job listing you can create a list of the links and append theme:

wait = WebDriverWait(driver, 10)
elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, '//*[@id="search-results-container"]//article/job/a')))
list_of_links = []
for i in elements:
    list_of_links.append(i.get_attribute('href'))
    # print(f"link = {i.get_attribute('href')}")
print(list_of_links)

Hope this helps you!