Python findAll没有在beautifulsoup 3上工作

Question

I am trying to parse a html file and write the results to a csv file. The html file is:

<table BORDER='1' CELLSPACING='0' CELLPADDING='0'>
<tr>
    <td><small>15</small></td >
    <td><small><small>Cat</small></small></td>
</tr>
<tr>
    <td><small><small>16</small></small></td>       
    <td><small><small>&nbsp;</small></small></td>       
</tr>
<tr>
    <td><small>17</small></td >
    <td><small><small>Dog</small></small></td>
</tr>
</table>

and the code I have atm is:

import csv
from BeautifulSoup import BeautifulSoup as bs

soup = bs (open("Animals.html"))
for i in soup.findAll('small'):
if " " in i.text:
    i.string = '-'

print soup

f = csv.writer(open("Animals.csv", "a"))   # Open the output file for writing before the loop

trs = soup.findAll('tr')

for tr in trs:

    tds = tr.findAll("td")

    try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
    id = str(tds[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
    animal = str(tds[1].get_text())

except:
    print "Bad tr string"
    continue #This tells the computer to move on to the next item after it encounters an error

f.writerow([id, animal])

When I print out the contents of soup after replacing the %nbsp; I get:

<table BORDER='1' CELLSPACING='0' CELLPADDING='0'>
<tr>
    <td><small>15</small></td >
    <td><small></small><small>Cat</small></td > 
</tr>
<tr>
    <td><small><small>16</small></small></td >      
    <td><small></small><small>-</small></td >       
</tr>
<tr>
    <td><small>17</small></td >
    <td><small></small><small>Dog</small></td >
</tr>
</table>

But when I look at the .csv file it is empty. However if I change the code to use BeautifulSoup 4, then I can't replace the   but the results will be saved to the .csv file. The other code that I use is:

import csv
from bs4 import BeautifulSoup as bs

soup = bs (open("Animals.html"))

f = csv.writer(open("Animals.csv", "w"))   # Open the output file for writing before the loop

trs = soup.find_all('tr')

for tr in trs:

tds = tr.find_all("td")

try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
    id = str(tds[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
    animal = str(tds[1].get_text())

except:
    print "Bad tr string"
    continue #This tells the computer to move on to the next item after it encounters an error

f.writerow([id, animal])

The reason why that one won't do me is because I want the   to be replaced with - and I haven't been able to get that (the find_all()) to work with beautifulsoup 4. What is causing the information to be saved to the csv file and how can I fix it (and/or get it working with beautifulsoup 4)?

Answer 1

BeautifulSoup 4 will convert that 'nbsp;' into a Unicode character \\xa0 when the 'soup' is constructed. If you search-and-replace on that unicode character it will work:

soup = BeautifulSoup(html)

for i in soup.find_all('small'):
    i.string.replace_with(i.string.replace(u'\xa0', '-'))

The syntax there is a little verbose. This is because i.string is not a string, but a bs4.element.NavigableString . You can't edit these in place with a straightforward i.string.replace(...) ; instead you must call beautifulsoup's own replace_with method.

replace_with accepts just one argument, so we have to generate the new version of the string and pass it in. For this we can use Python's built in replace method for strings, to strip out the u'\\xa0' characters and replace them with '-'

Alternatively, you could just use regular expressions on the original HTML. If all you need is to replace all   instances with a - :

import re

newhtml = re.sub(r' ', '-', html)

Though you could customise this further so it only affects <small> tags - let me know if you'd like this added to the answer.