Selenium Python unable to find web element

Question

I have been trying to extract posts from a forum found at this URL: https://www.thestudentroom.co.uk/showthread.php?t=7263973

The body of the text I am trying to extract is under:

<div class="post-content">

Yet I keep getting the following error whether I use get element to search by XPATH or CLASS_NAME:

NoSuchElementException

I have tried the following as well as looking at several of the similar posts on SO but can't find a solution that works for me, any help would be appreciated

options = Options()
options.add_argument("--headless")
options.headless = True

def get_posts(url):
    driver = webdriver.Chrome(options = options) 
    WebDriverWait(driver, 5)
    driver.get(url)                                                                            
#   posts = driver.find_element(By.XPATH, '/html/body/div[2]/div/div[6]/div[1]/div[1]/div[6]/div[3]/div[2]/div[2]').text 
    posts = driver.find_element(By.CLASS_NAME, 'post-content');
    return posts

SR_posts = get_posts(url = "https://www.thestudentroom.co.uk/showthread.php?t=7263973")
SR_posts

Edit: added picture to the HTML class 'post-content' that contains the text HTML of webpage

Edit 2: Second picture of inspect element Inspect element of text body

Answer 1

Here are a few things you can try:

Make sure that the element you are trying to find is actually present in the page's HTML source code. You can do this by right-clicking on the element in your browser and selecting "Inspect" or "View source."

Check that you are using the correct method to locate the element. For example, if you are using the find_element_by_class_name method, make sure that the class name you are using is spelled correctly and is unique to the element you are trying to locate.

make sure that the XPath you are using is correct and correctly identifies the element you are trying to locate.

Make sure that the element you are trying to locate is not being loaded dynamically via JavaScript after the page has finished loading. In this case, you may need to use a WebDriverWait to wait for the element to become available before attempting to locate it.

I hope this helps!

Answer 2

Try this to get the value of the post:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver


def get_posts(url):
    driver = webdriver.Chrome()
    driver.maximize_window()
    wait = WebDriverWait(driver, 5)
    driver.get(url)
    post = wait.until(EC.presence_of_element_located((By.XPATH, f"//div[@class='styles__PostContent-sc-1r7c0ap-3 kylDhV']/span")))
    return post

SR_post = get_posts(url = "https://www.thestudentroom.co.uk/showthread.php?t=7263973")
print(SR_post.text)

Advices:

• Wait always for the element you want to interact for
• Use EC (Expected conditions) to verify your element is accessible in the way you need (In my example is EC.presence_of_element_located but you could want something else like wait to be clickable etc.

I used those 2 advices in the code.