Python Selenium. Unable to locate elements

Question

I've been trying for hours to extract the text from an unknown number of elements on a website. I uploaded 2 pictures so you can see similarities of the elements to identify them. They have the same class name, for example. The black underlined text is basically the text I want to extract.

I haven't even come to the step of extracting the text because I'm not able to even find the element.

Here is one of the things I've tried:

a=driver.find_elements_by_class_name("pooVf")
print(a)

The result I got was '[]'.

I have also tried to use css selector

a=driver.find_elements_by_css_selector("a.pooVf.prDW")
print(a)

Also got '[]'.

I have no clue how to add the html code as text. Everytime I tried it didn't work.

Answer 1

Did you try to use the function " text "?

driver.find_elements_by_css_selector(".....").text

Answer 2

Have you tried using the element's XPath instead of its CSS selector or class name? I would've left this as a comment, but I don't have enough reputation.

Answer 3

Can you try using

driver.find_element_by_css_selector("a.pooVf.prDW")

Instead of

driver.find_elements_by_css_selector("a.pooVf.prDW")

Answer 4

Possibility: Dynamically added elements

Since the element exists when you manually inspect the DOM in a browser after a delay, but Selenium complains that it can't find it, I suspect that the element is being added dynamically by JavaScript. Selenium has tools for dealing with situations like these. See their docs on waits .

These days, most of the web apps are using AJAX techniques. When a page is loaded by the browser, the elements within that page may load at different time intervals. This makes locating elements difficult: if an element is not yet present in the DOM, a locate function will raise an ElementNotVisibleException exception. Using waits, we can solve this issue. Waiting provides some slack between actions performed - mostly locating an element or any other operation with the element.

One option is to add the following line to make all of the find methods wait for a few seconds before giving up:

driver.implicitly_wait(5) # seconds

Don't make the delay too long, because you don't want to be waiting forever if the element really does not exist. Using Selenium's wait functionality is better than time.sleep . There's no point waiting once the element exists, but sleep has no way of knowing when this happens, whereas with implicitly_wait , the code should return as soon as the element is available.

Possibility: Different session / cookies

Maybe the elements you are looking for depend on information (cookies, local storage, etc.) that is specific to your non-headless Chrome connection, and is not available to headless Chrome. It is like using two different browsers. Try using Chrome's Incognito mode and see if you can still find the elements when you inspect the DOM. If that reproduces the issue (the elements are missing), then that means you need to add additional lines to your Selenium code to replicate what you did in the browser: perhaps signing in with credentials on a login page, clicking some upload buttons, etc. All these are actions that need to be coded into your headless version - they can't depend on actions you manually performed in non-headless Chrome. Even then you may have difficulty with your selectors as written: pooVf , t3RpAe , prDW , and Rgstwe all look like fairly unique, unpredictable strings. If your sequence of actions (login, upload) results in different strings each time, then you will need to find these elements in a different way. Maybe one of their parent elements can be found more reliably.