如何在<span>带有特定 class 和特定文本的标签后面的 HTML 代码中获取下一行的字符串？</span>

Question

I'm trying to scrape out some product specifications from some e-commerce website. So I have a list of URLs to various products, I need my code to go to each (this part is easy) and scrape out the product specs I need. I have been trying to use ParseHub — it works for some links but it does not for other. My suspicion is, for example, 'Wheel diameter' changes its location every time so it ends up grabbing wrong spec value.

One of such parts, for example, in HTML looks like this:

<div class="product-detail product-detail-custom-field">
          <span class="product-detail-key">Wheel Diameter</span>
          <span data-product-custom-field="">8 Inches</span>
        </div>

What I think I could do is if I use BeautifulSoup and if I could somehow using smth like

if soup.find("span", class_ = "product-detail-key").text.strip()=="Wheel Diameter":
                *go to the next line and grab the string inside*

How can I code this? I really apologize if my question sounds silly, pardon my ignorance, I'm pretty new to webscraping.

Answer 1

You can use .find_next() function:

from bs4 import BeautifulSoup

html_doc = """
<div class="product-detail product-detail-custom-field">
  <span class="product-detail-key">Wheel Diameter</span>
  <span data-product-custom-field="">8 Inches</span>
</div>
"""

soup = BeautifulSoup(html_doc, "html.parser")

diameter = soup.find("span", text="Wheel Diameter").find_next("span").text
print(diameter)

Prints:

8 Inches

---

Or using CSS selector with + :

diameter = soup.select_one('.product-detail-key:-soup-contains("Wheel Diameter") + *').text

Answer 2

Using css selectors you can simply chain / combinate your selection to be more strict. In this case you select the <span> contains your string and use adjacent sibling combinator to get the next sibling <span> .

diameter = soup.select_one('.product-detail-key:-soup-contains("Wheel Diameter") + span').text

or

diameter = soup.select_one('span.product-detail-key:-soup-contains("Wheel Diameter") + span').text

Note: To avoid AttributeError: 'NoneType' object has no attribute 'text' , if element is not available you can check if it exists before calling text method:

diameter = e.text if (e := soup.select_one('.product-detail-key:-soup-contains("Wheel Diameter") + span')) else None

Example

from bs4 import BeautifulSoup

html_doc = """
<div class="product-detail product-detail-custom-field">
  <span class="product-detail-key">Wheel Diameter</span>
  <span data-product-custom-field="">8 Inches</span>
</div>
"""

soup = BeautifulSoup(html_doc, "html.parser")

diameter = e.text if (e := soup.select_one('.product-detail-key:-soup-contains("Wheel Diameter") + span')) else None

Answer 3

If you are using parsehub to collect the data:

<div class="product-detail product-detail-custom-field">
      <span class="product-detail-key">Wheel Diameter</span>
      <span data-product-custom-field="">8 Inches</span>
    </div>

and you are after the innerText under

      <span data-product-custom-field="">8 Inches</span>

Then what I would do is use a CSS selector to select the class of the first span. Place a '+' just infront of it and it will select the next sibling element.

such as:

.product-detail-key +

your result:

<span data-product-custom-field="">8 Inches</span>

Then all you have to do is choose to export the inner text, so under export type:

$e.text

This will scrape the following:

8 Inches