在多个位置使用find_elements_by_xpath

Question

Here is the HTML snippet:

<section class="node_category" id="kui_3_1515304072474_68">
    <h3 class="">User details</h3>
<ul class="" id="kui_3_1515304072474_67">
<li class="contentnode" id="kui_3_1515304072474_66">
<dl id="kui_3_1515304072474_65">
<dt class="">Country
</dt>
<dd class="" id="kui_3_1515304072474_64">United States
</dd>
</dl></li>
<li class="contentnode">
<dl>
<dt class="">City/town
</dt>
<dd class="">Somewhere
</dd>
</dl></li>
<li class="contentnode" id="kui_3_1515304072474_76">
<dl id="kui_3_1515304072474_75">
<dt class="">Company
</dt>
<dd class="" id="kui_3_1515304072474_74">ABC Inc
</dd>
</dl></li>
</ul></section>

I want to extract text from the following HTML class by XPath:

/ul/li[@class='contentnode'][3]/dl/dd

This "contentnode" class has multiple positions from 1 to maximum 6 for other pages. In this example, the maximum is 3. To designate all positions, I construct XPath like below:

//li[@class='contentnode'][1 <= position() and position() < 7]/dl/dd

Now, I plug into my Python code like below:

from selenium import webdriver


lst=[]
browser = webdriver.Chrome('./path')
url = "https://<target URL>"
browser.get(url)
contents = browser.find_elements_by_xpath("//li[@class='contentnode'][1 <= position() and position() < 7]/dl/dd")

for t in contents:

    lst.append([t.text])

print(lst)

However, the output only shows position 1. It should show all the text from the position 1 to 6.

[Edit] Also I tried,

//li[@class='contentnode'][contains(@id,'kui_3')]/dl/dd

but still does not work. It does not show any error but the result is nothing.

What's wrong with my code?

Answer 1

This is working code for your needs:

from selenium import webdriver


lst = []
browser = webdriver.Chrome()
browser.get("https://<target URL>")

contents = browser.find_elements_by_xpath("//li[@class='contentnode'][1 <= position() and position() < 7]/dl/dd")

for t in contents:

    lst.append(t.text)

print(lst)

browser.quit()

The result will be (according to your HTML):

['United States', 'Somewhere', 'ABC Inc']

Hope it helps you!

Answer 2

Try below code

from selenium import webdriver

lst=[]
browser = webdriver.Chrome('./path')
url = "https://<target URL>"
browser.get(url)
contents = browser.find_elements_by_xpath("//li[@class='contentnode']/dl/dd")
print len(contents)

for t in contents:
    lst.append(t.text)

print(lst)

Answer 3

Did you try with css selector? If not then you should give it a go:

for items in browser.find_elements_by_css_selector(".contentnode"):
    data = ' '.join([' '.join(item.text.split()) for item in items.find_elements_by_css_selector("dd")])
    print(data)