BeautifulSoup解析HTML表

Question

This is my first time using BeautifulSoup and I am trying to parse an HTML table. So far, through other examples, I have been able to write some simple code to get very close to what I need. However, by using the ele.text.strip() , I end up losing part of the information that I want to keep.

As seen below, here is what my code looks like now:

soup = BeautifulSoup(open("data_table.htm"))

table = soup.find("div", id="CT_Main_1_divResults")
table_body = table.find('tbody')
rows = table_body.find_all('tr')

data = []
for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append(cols)

Results:

[u'$4,090,000,000',
 u'13.61%',
 u'4,550,000',
 u'100 Grainger Pkwy.',
 u'',
 u'',
 u'']

I thought maybe I could just eliminate the ele.text.strip() line, and use the same code, as seen below:

data = []
for row in rows:
    cols = row.find_all('td')
    data.append(cols)

Here are the results that provides below:

[<td><span style="text-align: right; height: 36px;">$4,090,000,000</span></td>,
 <td><span style="text-align: right; height: 36px;">13.61%</span></td>,
 <td><span style="text-align: right; height: 36px;">4,550,000</span></td>,
 <td class=""><span style="text-align: right; height: 36px;">100 Grainger Pkwy.</span></td>,
 <td><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/True.gif"/></span></td>,
 <td><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/cancel.gif"/></span></td>,
 <td class="tdbrdrright"><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/True.gif"/></span></td>]

One way around this might be to use the second option and do some fancy string parsing to grab what I need, but I hope theres a better way. In the end, I want the data to look like below. How can I adjust my code to achieve this?

[u'$4,090,000,000',
 u'13.61%',
 u'4,550,000',
 u'100 Grainger Pkwy.',
 u'Inside%20the%20Databases.com_files/True.gif',
 u'Inside%20the%20Databases.com_files/calcel.gif',
 u'Inside%20the%20Databases.com_files/True.gif']

Answer 1

import bs4

html = '''<td><span style="text-align: right; height: 36px;">$4,090,000,000</span></td>,
 <td><span style="text-align: right; height: 36px;">13.61%</span></td>,
 <td><span style="text-align: right; height: 36px;">4,550,000</span></td>,
 <td class=""><span style="text-align: right; height: 36px;">100 Grainger Pkwy.</span></td>,
 <td><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/True.gif"/></span></td>,
 <td><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/cancel.gif"/></span></td>,
 <td class="tdbrdrright"><span style="text-align: right; height: 36px;"><img src="Inside%20the%20Databases.com_files/True.gif"/></span></td>'''
soup = bs4.BeautifulSoup(html, 'lxml')

for td in soup('td'):
    if td.text:
        print(td.text)
    else:
        print(td.img.get('src'))

out:

$4,090,000,000
13.61%
4,550,000
100 Grainger Pkwy.
Inside%20the%20Databases.com_files/True.gif
Inside%20the%20Databases.com_files/cancel.gif
Inside%20the%20Databases.com_files/True.gif

Change the print to append , and you will get a list of this output.

The missing info you want is in the img tag's attribute, not a text.

Answer 2

Give this a try. You'll need to adjust based on what you want to do if there are, say, multiple img tags, or text as well as img tags, etc., but this should get you started down the right path.

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("data-table.html"), 'html.parser')

table = soup.find("div", id="CT_Main_1_divResults")
table_body = table.find('tbody')
rows = table_body.find_all('tr')

data = []
for row in rows:
    cols = []
    for col in row.find_all('td'):
        t = col.text.strip()
        if not t:
            for img in row.find_all('img'):
                t = img.attrs['src']

        cols.append(t)
    data.append(cols)

print(data)

Output:

[[u'$4,090,000,000', u'13.61%', u'4,550,000', u'100 Grainger Pkwy.', u'Inside%20the%20Databases.com_files/True.gif', u'Inside%20the%20Databases.com_files/True.gif', u'Inside%20the%20Databases.com_files/True.gif']]