使用漂亮汤的Python Web Scraping HTML表

Question

This is my HTML Table.

<table class="table_c" id="myd">
<tbody>
    <tr class="grp">
        <th class="col>MyGrp1</th>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item0.1 Header</th>
        <td class="col data" data-th="MyGrp1">Item0.1 Value</td>
    </tr>
    <tr class="grp">
        <th class="col label" colspan="2" scope="row">MyGrp</th>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item1.1 Header</th>
        <td class="col data" >Item1.1 Value</td>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item1.2 Header</th>
        <td class="col data">Item1.2 Value</td>
    </tr>
    <tr class="item">
    <th class="col label" scope="row">Item1.3 Header</th>
    <td class="col data"">Item1.2 Value</td>
    </tr>
</tbody>
</table>

I want the table to parsed as below

MyGrp1<new line>
<tab char>Item0.1 Header<tab char>Item0.1 Value<new line>
MyGrp2<new line>
<tab char>Item1.1 Header<tab char>Item1.1 Value<new line>
<tab char>Item1.2 Header<tab char>Item1.2 Value<new line>
<tab char>Item1.3 Header<tab char>Item1.3 Value<new line>

I can get all the nodes of 'tr' or 'th'. But I don't know how to iterate the table node by node. How can I scrape the Html table and get my above result?

Answer 1

i used pandas for this

import pandas as pd
import html5lib

string="""<table class="table_c" id="myd">
<tbody>
    <tr class="grp">
        <th class="col">MyGrp1</th>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item0.1 Header</th>
        <td class="col data" data-th="MyGrp1">Item0.1 Value</td>
    </tr>
    <tr class="grp">
        <th class="col label" colspan="2" scope="row">MyGrp</th>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item1.1 Header</th>
        <td class="col data" >Item1.1 Value</td>
    </tr>
    <tr class="item">
        <th class="col label" scope="row">Item1.2 Header</th>
        <td class="col data">Item1.2 Value</td>
    </tr>
    <tr class="item">
    <th class="col label" scope="row">Item1.3 Header</th>
    <td class="col data"">Item1.2 Value</td>
    </tr>
</tbody>
</table>"""
df = pd.read_html(string)
print(df)

output

[                0              1
0          MyGrp1            NaN
1  Item0.1 Header  Item0.1 Value
2           MyGrp            NaN
3  Item1.1 Header  Item1.1 Value
4  Item1.2 Header  Item1.2 Value
5  Item1.3 Header  Item1.2 Value]

Answer 2

But I don't know how to iterate the table node by node.

BeautifulSoup 's find_all provides you with a sequence of tag objects that you can loop through.

Also please note that your html table has synthax problems: <th class="col>MyGrp1</th> - missing quote <td class="col data"">Item1.2 Value</td> - double quotes

So provided that sample is your html table as a sting and it has valid html here's a sample of what you could do:

from bs4 import BeautifulSoup as bs

soup = bs(sample, 'lxml-html')
trs = soup.find_all('tr')
group = None # in case there are items before the first group
for tr in trs:
    if 'grp' in tr.get('class'):
        print(tr.th.text)
    elif 'item' in tr.get('class'):
        label = tr.th.text
        value = tr.td.text
        print('{} {}'.format(label, value))

Answer 3

I did the following to get the answer. I give my solution here. Please correct me If I am wrong.

result = ""
for tr in table_t.findAll('tr'):
    if 'grp' in tr.get("class"):
        for th in tr.findAll('th'):
            result += "\n" + th.text.strip()
            #print(th.text.strip())
    elif 'item' in tr.get("class"):
        children_th = tr.find("th")
        children_td = tr.find("td")
        result += "\n\t" + children_th.text.strip() + "\t" + children_td.text.strip()
print(result)