如何使用Selenium和Python从div中收集这些数据

Question

I have been using Selenium and Python to scrape a webpage and I am having difficulty collecting data that I want out of a div that has the following structure:

<div class="col span_6" style="margin-left: 12px;width: 47% !important;">
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Heading1</span>
    <span class="MainGridcolumn2">Text that I want</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Another heading</span>
    <span class="MainGridcolumn2">More text that I want</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Next heading</span>
    <span class="MainGridcolumn2">Even more text</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Yet another heading</span>
    <span class="MainGridcolumn2">Piece of text</span>
  </div>
</div>

The div has a number of rows, each with 2 columns containing the data/text inside of span tags. There are no CSS ids.

I'm only interested in collecting the text contained within the 'MainGridcolumn2' span classes.

I've tried the below to navigate to the first heading, with the intention of then trying to use 'following_sibling' to move down to the next span tag containing the text, but I can't even get this to work as it isn't returning any text when I try to print it to the console:

driver.find_element_by_xpath("//span['@class=MainGridcolumn1'][contains(text(), 'Heading1')]").text

and

driver.find_element_by_xpath("//span[contains(text(), 'Heading1')]").text

Answer 1

One way would be to get the the enclosing div ie the grandparent and pull the spans from that:

h = """<div class="col span_6" style="margin-left: 12px;width: 47% !important;">
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Heading1</span>
    <span class="MainGridcolumn2">Text that I want</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Another heading</span>
    <span class="MainGridcolumn2">More text that I want</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Next heading</span>
    <span class="MainGridcolumn2">Even more text</span>
  </div>
  <div class="MainGridRow">
    <span class="MainGridcolumn1">Yet another heading</span>
    <span class="MainGridcolumn2">Piece of text</span>
  </div>
</div>

  <div class="MainGridRow">
    <span class="MainGridcolumn1">Yet another heading</span>
    <span class="MainGridcolumn2">Piece of text I don't want</span>
  </div>"""

from lxml import html

xm = html.fromstring(h)
div = xm.xpath("//span[@class='MainGridcolumn1'][contains(text(), 'Heading1')]/../..")[0]
print(div.xpath(".//span[@class='MainGridcolumn2']/text()"))

Which would give you:

['Text that I want', 'More text that I want', 'Even more text', 'Piece of text']

You could also just select the parent and get the parents siblings

from lxml import html

xm = html.fromstring(h)
div = xm.xpath("//span[@class='MainGridcolumn1'][contains(text(), 'Heading1')]/..")[0]
print(div.xpath(".//span[@class='MainGridcolumn2']/text() | .//following-sibling::div/span[@class='MainGridcolumn2']/text()"))