美丽的汤对象省略信息

Question

Problem: The beautiful soup object seems to delete valuable information from the HTML. Why is it doing this, and how can I extract this field?

Example: The raw HTML I'm interested in expresses this:

<div id="KittyChow">
            <h4 class="noteText">foodAmount</h4>
            <span>< 1 tsp</span>
        </div>

When I create my soup object however, the corresponding lines of HTML become:

<div id="KittyChow"><h4 class="noteText">foodAmount</h4><span></span></div>

My problem and question: Why has it deleted the information in between span and /span? Is it because the "less than/ <" sign indicated some HTML so it stripped that? I want to know WHY this happens. I couldn't seem to find an explanation in the documentation.... Is there ANY WAY to parse this in BeautifulSoup?

Second: How do I extract this < 1 tsp parameter? I've tried creating a regex with a left and right endpoint, and that ALMOST works. I know how to use regex to return text if I specify a "left substring match" and a "right substring match." For instance, the code below will return "cat."

import re

string= "The cat is obese."
left= "The"
right= "is obese."

pattern= re.compile(left + "(.*?)" + right)
answer= pattern.findall(string)[0]

print answer

The issue is, when I replace the left and right match string with HTML, I get the "index is out of bounds" error, because of the whitespace and indentation implicated with casting HTML into a string.

So as you can tell... I've done a fair bit of research, and I'm still stuck on extracting < and > signs within fields/ attributes of HTML tags using both BeautifulSoup and Python's regex module. Please help me? :)

Answer 1

Do you have control over your html? It is malformed. Instead of

<div id="KittyChow">
    <h4 class="noteText">foodAmount</h4>
    <span>< 1 tsp</span>
</div>

It should look like

<div id="KittyChow">
    <h4 class="noteText">foodAmount</h4>
    <span>&lt; 1 tsp</span>
</div>

http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references

If you are generating the html on the server-side, it should be easy in any language to encode your entities: php python ruby

edit : According to this other answer: https://stackoverflow.com/a/14171433/1253312 You can do this:

BeautifulSoup("<div> < 20 </div>", "html5lib")

Which tells BS to use a different parser, which can handle the < character.

Answer 2

The HTML is broken. You can't have an unescaped < character in HTML; the parser will get mightily confused. As a workaround, in this particular example you could replace < followed by a space with < followed by a space:

raw_html = raw_html.replace("< ", "&lt; ")

Although this is not a general solution.