How to extract or Scrape data from HTML page but from the element itself

Question

Currently i use lxml to parse the html document to get the data from the HTML elements but there is a new challenge, there is one data stored as ratings inside HTML elements

https://i.stack.imgur.com/bwGle.png

<p data-rating="3">
                                <span class="glyphicon glyphicon-star xh-highlight"></span>
                                <span class="glyphicon glyphicon-star xh-highlight"></span>
                                <span class="glyphicon glyphicon-star xh-highlight"></span>
                            </p>

Its easy to extract text between tags but within tags no ideas. What do you suggest?

Challenge i want to extract "3" URL: https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops

Br, Gabriel.

Answer 1

Try below script:

from bs4 import BeautifulSoup
import requests

BASE_URL = "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops"

html = requests.get(BASE_URL).text
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all("div", {"class":"ratings"}):
    # get all child from the tags
    for h in tag.children:
        # convert to string data type
        s = h.encode('utf-8').decode("utf-8") 

        # find the tag with data-rating and get text after the keyword
        m = re.search('(?<=data-rating=)(.*)', s)

        # check if not None
        if m:
            #print the text after data-rating and remove last char
            print(m.group()[:-1])

Answer 2

If I understand your question and comments correctly, the following should extract all the rating in that page:

import lxml.html
import requests

BASE_URL = "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops"

html = requests.get(BASE_URL)
root = lxml.html.fromstring(html.text)
targets = root.xpath('//p[./span[@class]]/@data-rating')

For example:

targets[0]

output

3