将html表转换为字典而不丢失结构

Question

I'm new to python (and programming) and using BeautifulSoup for the first time.

I'm trying to find the best way to parse the contents of a table in html and convert to a dictionary - ideally in the least brittle way.

Here is an example of the HTML I'm trying to parse (I've put key value numbers for the text I'm trying to pick up).

<div class="tablename">
<table border="0" cellpadding="0" cellspacing="0" style="border: 1px solid #dddddd;  border-collapse: collapse; font-family: Arial, Helvetica, sans-serif; font-size: 14px; margin: 0; padding: 0; width: 100%">
<thead>
<tr>
<th colspan="4" style="background-color: #000; border: 1px solid #616161; color: #ffffff; font-size: 14px; font-weight: bold; line-height: 20px; padding: 14px 20px 12px 20px; text-align: left">Some text not needed</th>
</tr>
</thead>
<tbody>
<tr>
<td style="width: 20px"> </td>
<td style="border-bottom: 1px solid #dddddd; color: #666666; font-size: 14px; line-height: 20px; padding: 11px 20px 10px 0; text-align: left; width: 42.5%; vertical-align: middle">Key 1</td>
<td style="border-bottom: 1px solid #dddddd; color: #000; font-size: 14px; line-height: 20px; padding: 11px 0 10px 0; text-align: left; vertical-align: middle">Value 1</td>
<td style="width: 20px"> </td>
</tr>
<tr>
<td> </td>
<td style="border-bottom: 1px solid #dddddd; color: #666666; font-size: 14px; line-height: 20px; padding: 11px 20px 10px 0; text-align: left; vertical-align: middle">Key 2</td>
<td style="border-bottom: 1px solid #dddddd; color: #000; font-size: 14px; line-height: 20px; padding: 11px 0 10px 0; text-align: left; vertical-align: middle">Value 2</td>
<td> </td>
</tr>
<tr>
<td> </td>
<td style="border-bottom: 1px solid #dddddd; color: #666666; font-size: 14px; line-height: 20px; padding: 11px 20px 10px 0; text-align: left; vertical-align: middle">Key 3</td>
<td style="border-bottom: 1px solid #dddddd; color: #000; font-size: 14px; line-height: 20px; padding: 11px 0 10px 0; text-align: left; vertical-align: middle">Value 3</td>
<td> </td>
</tr>
<tr>

And the code I'm using:

import requests
from bs4 import BeautifulSoup

html = requests.get('https://examplewebaddress.com')
soup = BeautifulSoup(html.text)
print(soup.tbody.text)

I could then loop over the soup.tbody.text string and split this into key value pairs. But this doesn't seem to be a good way and I seem to be losing the structure of the table by converting it to a string and then building that back up again into a dictionary.

Is there a more direct way to parse a table with BeautifulSoup (or something more suitable) into a dictionary which I can then use?

Answer 1

The idea is to iterate over table rows and for each row extract the text of the second and the third cells that would represent key and a value of the future dictionary:

soup = BeautifulSoup(html.text)

result = dict([[item.get_text(strip=True) for item in row.find_all('td')[1:3]]
               for row in soup.select("div.tablename table tr")[1:]])

print result

For the provided sample data, it prints:

{u'Key 1': u'Value 1', u'Key 2': u'Value 2', u'Key 3': u'Value 3'}

div.tablename table tr is a CSS selector that would match all tr elements under table element that has div with a class="tablename" as a parent. We are slicing the result of select ( [1:] ) to skip the first header row.