Python Selenium - Find element by class and text

Question

I'm trying to paginate through the results of this search: Becoming Amazon search . I get a 'NoSuchElementException'..'Unable to locate element: < insert xpath here >

Here is the html:

<div id="pagn" class="pagnHy">
    <span class="pagnLink">
        <a href="/s/ref=sr_pg_2?rh=...">2</a>
    </span>
</div>

Here are the xpaths I've tried:

driver.find_element_by_xpath('//*[@id="pagn" and @class="pagnLink" and text()="2"]')

driver.find_element_by_xpath('//div[@id="pagn" and @class="pagnLink" and text()="2"]')

driver.find_element_by_xpath("//*[@id='pagn' and @class='pagnLink' and text()[contains(.,'2')]]")

driver.find_element_by_xpath("//span[@class='pagnLink' and text()='2']")

driver.find_element_by_xpath("//div[@class='pagnLink' and text()='2']")

If I just use find_element_by_link_text(...) then sometimes the wrong link will be selected. For example, if the number of reviews is equal to the page number I'm looking for (in this case, 2), then it will select the product with 2 reviews, instead of the page number '2'.

Answer 1

You're trying to mix attributes and text nodes from different WebElements in the same predicate. You should try to separate them as below:

driver.find_element_by_xpath('//div[@id="pagn"]/span[@class="pagnLink"]/a[text()="2"]')

Answer 2

When I look at the markup, I'm seeing the following:

<span class="pagnLink">
    <a href="/s/ref=sr_pg_2?rh=...">2</a>
</span>

So you want to find a span with class pagnLink that has a child a element with the text 2 , or:

'//*[@class="pagnLink"]/a[text()="2"]'

Answer 3

Sometimes it might be better to take a intermediate step and first to get the element which contains the results. Afterwards you just search within this element. Doing it this way you simplify your search terms.

from selenium import webdriver

url = 'https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Daps&fieldkeywords=becoming&rh=i%3Aaps%2Ck%3Abecoming'
driver = webdriver.Firefox()
resp = driver.get(url)
results_list_object = driver.find_element_by_id('s-results-list-atf')
results = results_list_object.find_elements_by_css_selector('li[id*="result"]')

for number, article in enumerate(results):
    print(">> article %d : %s \n" % (number, article.text))