Python Selenium trying to scrape some text information from a website and save it into a list

Question

Having an issue writing code with Selenium in python to scrape information from a depop listing for a personal project of mine. Specifically the title, price, condition, size and the brand.

Ive used selenium and the error it keeps giving me is no such element: Unable to locate element:. So kinda stuck at a dead end in terms of extracting the text and saving it to a list. Any advice?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time
import requests
from bs4 import BeautifulSoup

# specify the URL you want to scrape
url = 'https://www.depop.com/products/_realvintage-vintage-south-beach-florida-embroidered/'
# results = requests.get(url)

chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome("D:\\Selenium_python2\\chromedriver.exe", chrome_options=chrome_options)

driver.get(url)
time.sleep(2)

driver.find_element('xpath', '//*[@id="__next"]/div/div[3]/div[2]/button[2]').click()

time.sleep(1)

element = driver.find_element('xpath', '//*[@id="main"]/div[1]/div[3]/div/div[1]/p/text()')
print(element)
text = element.text

print(text)

Answer 1

Please check if it is inside any iframe or shadow root .

If there is any iframe involved then you need to switch to that iframe and then use get_element .

driver.switchTo().frame(“<id of iframe>”)

This is a good blog to understand about Shadow DOM https://medium.com/@titus_on_testing/shadow-dom-in-python-selenium-df8d731b5005