如何读取HTML表格数据w美汤？ 返回“无”

Question

I'm using Beautiful Soup to read data from a HTML table. Why am I not getting a result from the table and how do I fix it? My code returns 'None'.

I see that there is javascript in the page source and have read that might be an issue. The url runs a report that is inputted into the table.

I've used soup.prettify() to check the HTML and it doesn't seem to give me the full source code. I'm unsure if this is an issue.

Here's the HTML of the table and the first data row:

    <table data-toggle="table"
        data-show-columns="true"
        data-show-export="true"
        data-show-toggle="true"
        class="table-data">
        <thead>
            <tr>
                <th data-field="RouteId" data-sortable="true">Route ID</th>
                <th data-field="RouteName" data-sortable="true">Route Name</th>
                <th data-field="TripId" data-sortable="true">Trip ID</th>
                <th data-field="TripName" data-sortable="true">Trip Name</th>
                <th data-field="InstanceId" data-sortable="true">INST ID</th>
                <th data-field="InstanceDate" data-sortable="true">INST Date</th>
                <th data-field="InstanceStatus" data-sortable="true">INST Status</th>
                <th data-field="InstanceCapacity" data-sortable="true">INST Cap.</th>
                <th data-field="NumOrders" data-sortable="true">Num. ORDs</th>
                <th data-field="OrderId" data-sortable="true">ORD ID</th>
                <th data-field="OrderType" data-sortable="true">ORD Type</th>
                <th data-field="OrderStatus" data-sortable="true">ORD Status</th>
                <th data-field="VehicleYear" data-sortable="true">VEH Year</th>
                <th data-field="VehicleMake" data-sortable="true">VEH Make</th>
                <th data-field="VehicleModel" data-sortable="true">VEH Model</th>
                <th data-field="VehicleRefNo1" data-sortable="true">VEH RefNo1</th>
                <th data-field="vehicleVin" data-sortable="true">VEH Vin</th>
                <th data-field="DriverId" data-sortable="true">DRV ID</th>
                <th data-field="DriverName" data-sortable="true">DRV Name</th>
                <th data-field="ScheduledPickupDateTime" data-sortable="true">Sch. Pick</th>
                <th data-field="ActualPickupPickupDateTime" data-sortable="true">Act. Pick</th>
                <th data-field="DeliveredDateTime" data-sortable="true">Hand. Rec.</th>
                <th data-field="HandheldDateTime" data-sortable="true">Del.</th>
            </tr>
        </thead>
        <tbody>

            <tr>
                <td>160</td>
                <td>8 LEG: MEM to PRES</td>
                <td>187</td>
                <td>Trip 1 - Leg 7</td>
                <td>740685</td>
                <td>2017-02-01</td>
                <td>Active</td>
                <td>9.00000</td>
                <td>9</td>
                <td>9110734</td>
                <td>LoadLegChild</td>
                <td>InRoute</td>
                <td>2015</td>
                <td>Jeep</td>
                <td>Patriot</td>
                <td>2000047350</td>
                <td>1C4NJPFBXFD318536</td>
                <td>1</td>
                <td>User, System</td>
                <td>2017-02-01 02:05 AM</td>
                <td>2017-02-01 02:20 AM</td>
                <td></td>
                <td></td>
            </tr>

Here is my attempt with Beautiful Soup:

from urllib.request import urlopen
from bs4 import BeautifulSoup

page = urlopen(url)
soup = BeautifulSoup(page,'lxml')
print(soup.find('table',{'class':'table-data table'}))

I've also tried xpath but received an empty list:

import requests
from lxml import html
NewPage = requests.get(url)
tree = html.fromstring(NewPage.content)
tree.xpath('//*[@id="content"]/div[2]/div[2]/div[2]/div[2]/table/tbody/tr[1]/td[1]')

UPDATE: I'm thinking the table I'm trying to use is dynamically created; how would I change my code to account for this? I've also tried using find_all to check my work but it doesn't bring back every table in the HTML, only the first one. Why is this?

page = requests.get(url)
pageText = page.text
soup = BeautifulSoup(pageText,'lxml')
print(soup.find_all('table'))

Here's the output:

[<table cellpadding="0" cellspacing="0" id="Login1">
<tr>
<td>
<div class="row">
<div class="col-md-6">
<div class="form-group">
<label for="UserName">Username</label>
<input class="form-control" id="Login1_UserName" name="Login1$UserName" type="text"/>
</div>
</div>
<div class="col-md-6">
<div class="form-group">
<label for="Password">Password</label>
<input class="form-control" id="Login1_Password" name="Login1$Password" type="password"/>
</div>
</div>
</div>
<div class="row">
<div class="col-md-6">
<input id="Login1_RememberMe" name="Login1$RememberMe" type="checkbox"/><label for="Login1_RememberMe">Remember my login</label>
</div>
<div class="col-md-6 text-right">
<input class="btn btn-default" id="Login1_Login" name="Login1$Login" type="submit" value="Login"/>
</div>
</div>
<p>
</p>
</td>
</tr>
</table>]

Answer 1

It looks to me like you are mixing up the old form used in earlier versions of beautiful soup and the newer.

I would try: soup.find("table", class_="table-data")

This is the syntax for the newer versions of beautiful soup. Hopefully, that is what you are using.

I don't have beautiful soup installed so I can't verify, but you could give that a try.

Answer 2

You have an error in your find call.

You are searching for a table element that have both table-data and table classes. But, as you can see, the table only has the class table-data , not table one. Replace your code with:

print(soup.find('table',{'class':'table-data'}))

UPDATE : It seems that the webpage, as you said in the update, is dynamically created. So please print the full HTML webpage (or save it to a file) and work around that code ( don't use the code you see in Google Chrome or other browser inspector , they have some code generated AFTER document loads.).

• If you have everything you need with that code, that's all.
• If you don't have what you need, please consider using Ghost webkit web client, instead of urllib/requests, to get the webpage HTML dynamically created. Then you can use pure JavaScript to get the element you are searching for, or use Beautiful Soup as well.