How can I iterate through tags using Python?

Question

I would like to iterate through some html and store data into a dictionary. Each iteration starts with:

<h1 class="docDisplay" id="docTitle">

I have the following code:

html = '<html><body><h1 class="docDisplay" id="docTitle">Data1</h1><p>other data<\p><h1 class="docDisplay" id="docTitle">Data2</h1><p>other data2<\p></html>'

soup=BeautifulSoup(html)
newdoc = soup.find('h1', id="docTitle")
title = newdoc.findNext(text=True)
data = title.findAllNext('p',text=True)
data_dict = {}
data_dict[title] = {'data': data}
print data_dict

Right now, the output is

{u'Data1': {'data': [u'other data<\\p>', u'Data2', u'other data2<\\p>']}}

I would like the output to be:

{u'Data1': {'data': [u'other data<\\p>']}, u'Data2': {'data': [u'other data2<\\p>']}}

I can't figure out how to start again once I reach a new h1 tag. Any ideas?

Answer 1

To match the text of the paragraphs under each header, I would try something like this (you may have to modify this depending on the exact output format that you want):

    from BeautifulSoup import BeautifulSoup

    html = """ 
    <html>
    <head>
    </head>

    <body>
      <h1 class="docDisplay" id="docTitle">Data1</h1>
      <p>other data</p>
      <p>Another paragraph under the first heading.</p>
      <h1 class="docDisplay" id="docTitle">Data2</h1>
      <p>other data2</p>
      <div><p>This paragraph is NOT a sibling of the header</p></div>
    </body>
    </html>
"""

soup = BeautifulSoup(html)

data_dict = {}
stuff_under_current_heading = []

firstHeader = soup.find('h1', id="docTitle")
for tag in [firstHeader] + firstHeader.findNextSiblings():
    if tag.name == 'h1':
        stuff_under_current_heading = []
        # I chose to strip excess whitespace from the header name:
        data_dict[tag.string.strip()] = {'data': stuff_under_current_heading}
        # Modifying the list modifies the value in the dictionary.
    # Take every <p> tag encountered between here and the next heading
    # and associate it with the most recently-seen <h1> tag.
    elif tag.name == 'p':
        stuff_under_current_heading.append(tag.string)
    # Include <p> tags that are not siblings of the <h1> tag but
    # are still part of the content under the header.
    else:
        stuff_under_current_heading.extend(tag.findAll('p', text=True))

print data_dict

This outputs

{u'Data1': {'data': [u'other data', u'Another paragraph under the first heading.']},   
 u'Data2': {'data': [u'other data2', u'This paragraph is NOT a sibling of the header']}}

Answer 2

@samplebias: @Lynch is right. If the OP doesn't close his/her tags properly, they simply cannot expect for the parser to be able to read their mind.

Try fixing your HTML, it will probably work then. =)

How can I iterate through tags using Python?

Question

2 answers

solution1
2 ACCPTED 2011-04-29 00:33:19

solution2
-1 2011-04-27 23:41:29

How can I iterate through tags using Python?

Question

2 answers

solution1 2 ACCPTED 2011-04-29 00:33:19

solution2 -1 2011-04-27 23:41:29

solution1
2 ACCPTED 2011-04-29 00:33:19

solution2
-1 2011-04-27 23:41:29