从文本中提取HTML标签和数据

Question

I'm using Python 2.7 to try and do a simple call to a website to extract the HTML data, which I've managed with the code below.

import requests
from HTMLParser import HTMLParser

name = "Mark"
surname = "Jacobs"

def req_getPageHTML(nume, prenume):
    url = "http://sample.com/page.aspx&Name=" + name + "&surname=" + surname
    response = requests.get(url).text
    return response

page_code = req_getPageHTML(nume, prenume)

htmlp = HTMLParser()

print htmlp.feed(page_code)

The next thing that I want to do is somehow extract or parse this UNICODE response ( print type(page_code) returns UNICODE ) to somehow extract some information from it.

Specifically, from the below sample HTML which I can get back, I want to extract the values (numbers which are slightly inset in the below HTML code and also prefixed with a > - this doesn't exist in the HTML code, it's just for being easily identified by you guys).

...
<tr class="tr1" OnClick="lockBac();">
    <td class="tdB" rowspan="2" nowrap="nowrap">1</td>
    <td class="tdB" rowspan="2" nowrap="nowrap">Jacobs D <br/>Mark</td>
    <td class="tdB" rowspan="2" align="Center">Math speciality</td>
    <td class="tdB" rowspan="2" align="Center">Advanced User</td>
        >   <td class="tdB" rowspan="2" align="Center">6.95</td>
        >   <td class="tdB" rowspan="2" align="Center">7.9</td>
        >   <td class="tdB" rowspan="2" align="Center">7.9</td>
    <td class="tdB" colspan="4" align="Center"></td>
    <td class="tdB" rowspan="2" align="Center">English</td>
    <td class="tdB" rowspan="2" align="Center">B2-B2-B2-B2-B2</td>
    <td class="tdB" colspan="3" align="Center">Mathematics MATH-INFO</td>
    <td class="tdB" colspan="3" align="Center">Informatics</td>
    <td bgcolor="lightgreen" class="tdB" rowspan="2" align="Center"></td>
    <td class="tdB" rowspan="2" align="Center">8.88</td>
    <td class="tdB" rowspan="2" align="Center">Success</td>
</tr>
<tr class="tr1" OnClick="lockBac();">
    <td class="tdB"></td>
    <td class="tdB"></td>
    <td class="tdB"></td>
    <td class="tdB"></td>
        >    <td class="tdB">9.35</td>
        >    <td class="tdB"></td>
        >    <td class="tdB">9.35</td>
        >    <td class="tdB">9.4</td>
    <td class="tdB"></td>
        >    <td class="tdB">9.4</td>
</tr>
...

What these numbers represent is Exam scores, which I will later put in a DB.

Now, I'm trying to look for an efficient way to extract these numbers as I would prefer to leave parsing the text to look for each element (manually with SUBSTR and so on) as a last option.

I did come across HTMLParser, which as you can see is also imported into my code, but the bottom print returns None .

I was under the impression that I can use this library to parse the text received from response and there would be an easier way to specify a tag ID (or something similar) and extract the relevant information from it (like it is shown in the HTMLParser examples section ), but I can't get the necessary information I want from using the feed method.

Maybe I'm not understanding this correctly and maybe I'm not using the appropriate tool, so that is why I also explained my goal.

I would appreciate any help in correcting or pointing me into the right direction.

Answer 1

Not sure how to work with what you have tried, but I have a different method.

You can grab lxml , a python library that helps out with scraping XML and HTML. It seems Requests will also help out with this project.

page = requests.get('http://www.example.com')
tree = html.fromstring(page.text)

The tree variable now contains all of the html document, which you can parse however you wish. Using Xpath would have something like

scores = tree.xpath('//td[@class="tdB"]/text()')

Hope that helps.

source