美丽的汤和搜索结果

Question

These are my first steps with python, please bear with me.

Basically I want to parse a Table of Contents from a single Dokuwiki page with Beautiful Soup. The TOC looks like this:

<div id="dw__toc">
<h3 class="toggle">Table of Contents</h3>
<div>

<ul class="toc">
<li class="level1"><div class="li"><a href="#section">#</a></div>
<ul class="toc">
<li class="level2"><div class="li"><a href="#link1">One</a></div></li>
<li class="level2"><div class="li"><a href="#link2">Two</a></div></li>
<li class="level2"><div class="li"><a href="#link3">Three</a></div></li>

I would like to be able to search in the content of the a-tags and if a result is found return its content and also return the href-link. So if I search for "one" the result should be

One
#link1

What I have done so far:

#!/usr/bin/python2

from BeautifulSoup import BeautifulSoup
import urllib2


#Grab and open URL, create BeatifulSoup object
url = "http://www.somewiki.at/wiki/doku.php"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
#Grab Table of Contents
grab_toc = soup.find('div', {"id":"dw__toc"})

#Look for all divs with class: li
ftext = grab_toc.findAll('div', {"class":"li"})
#Look for links
links = grab_toc.findAll('a',href=True)

#Iterate
for everytext in ftext:
    text = ''.join(everytext.findAll(text=True))
    data = text.strip()
    print data

for everylink in links:
    print everylink['href']

This prints out the data I want but I'm kind of lost to rewrite it to be able to search within the result and only return the searchterm. Tried something like

if data == 'searchtearm':
        print data
        break
else:
        print 'Nothing found'

But this is kind of a weak search. Is there a nicer way to do this? In my example the Beatiful Soup resultset is changed into a list. Is it better to search in the result set in the first place, if so then how to do this?

Answer 1

Instead of searching through the links one-by-one, have BeautifulSoup search for you, using a regular expression:

import re

matching_link = grab_toc.find('a', text=re.compile('one', re.IGNORECASE))

This would find the first a link in the table of contents with the 3 characters one in the text somewhere. Then just print the link and text:

print matching_link.string
print matching_link['href']

Short demo based on your sample:

>>> from bs4 import BeautifulSoup
>>> import re
>>> soup = BeautifulSoup('''\
... <div id="dw__toc">
... <h3 class="toggle">Table of Contents</h3>
... <div>
... 
... <ul class="toc">
... <li class="level1"><div class="li"><a href="#section">#</a></div>
... <ul class="toc">
... <li class="level2"><div class="li"><a href="#link1">One</a></div></li>
... <li class="level2"><div class="li"><a href="#link2">Two</a></div></li>
... <li class="level2"><div class="li"><a href="#link3">Three</a></div></li>
... </ul></ul>''')
>>> matching_link = soup.find('a', text=re.compile('one', re.IGNORECASE))
>>> print matching_link.string
One
>>> print matching_link['href']
#link1

In BeautifulSoup version 3, the above .find() call returns the contained NavigableString object instead. To get back to the parent a element, use the .parent attribute:

matching_link = grab_toc.find('a', text=re.compile('one', re.IGNORECASE)).parent
print matching_link.string
print matching_link['href']