使用Beautifulsoup和Python刮取复杂表
[英]Scraping a complex table using Beautifulsoup and Python
问题描述
<table cellspacing="0" rules="all" border="1" id="MainContent_grdUsers2" style="border-style:None;width:100%;border-collapse:collapse;">
<tbody><tr class="listHeader">
<th scope="col" style="width:11%;">Name</th><th scope="col" style="width:12%;">Password</th><th scope="col" style="width:16%;">Rights</th><th scope="col" style="width:10%;">Bureaus</th><th scope="col" style="width:15%;">FullName</th><th scope="col" style="width:16%;">Email</th><th scope="col" style="width:12%;">Status</th><th scope="col" style="width:12%;">Logon Tries</th>
</tr><tr>
<td>user1</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl02$txtManageUsersPassword" type="text" maxlength="50" id="MainContent_grdUsers2_txtManageUsersPassword_0" style="width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%; cursor: auto;" autocomplete="off">
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl02$ddlManageUsersRights" id="MainContent_grdUsers2_ddlManageUsersRights_0" style="width:95%;">
<option value="User">User</option>
<option selected="selected" value="Supervisor">Supervisor</option>
<option value="Administrator">Administrator</option>
<option value="Child Supervisor">Child Supervisor</option>
</select>
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl02$ddlManageUsersBureaus" id="MainContent_grdUsers2_ddlManageUsersBureaus_0" style="width:95%;">
<option value="255">High</option>
<option selected="selected" value="128">Medium</option>
<option value="0">Low</option>
</select>
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl02$txtManageUsersFullName" type="text" value="First1 Last1" maxlength="50" id="MainContent_grdUsers2_txtManageUsersFullName_0" style="width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%; cursor: auto;" autocomplete="off">
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl02$txtManageUsersEmail" type="text" value="user1@company.com" maxlength="50" id="MainContent_grdUsers2_txtManageUsersEmail_0" style="width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%; cursor: auto;" autocomplete="off">
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl02$ddlManageUsersStatus" id="MainContent_grdUsers2_ddlManageUsersStatus_0" style="width:95%;">
<option value="Active">Active</option>
<option selected="selected" value="Inactive">Inactive</option>
<option value="Terminated">Terminated</option>
</select>
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl02$txtManageUsersLogonTries" type="text" value="0" maxlength="1" id="MainContent_grdUsers2_txtManageUsersLogonTries_0" style="width:95%;">
</td>
</tr><tr style="background-color:#CED6E7;">
<td>user2</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl03$txtManageUsersPassword" type="text" maxlength="50" id="MainContent_grdUsers2_txtManageUsersPassword_1" style="background-color: rgb(206, 214, 231); width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%;" autocomplete="off">
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl03$ddlManageUsersRights" id="MainContent_grdUsers2_ddlManageUsersRights_1" style="background-color:#CED6E7;width:95%;">
<option value="User">User</option>
<option selected="selected" value="Supervisor">Supervisor</option>
<option value="Administrator">Administrator</option>
<option value="Child Supervisor">Child Supervisor</option>
</select>
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl03$ddlManageUsersBureaus" id="MainContent_grdUsers2_ddlManageUsersBureaus_1" style="background-color:#CED6E7;width:95%;">
<option value="255">High</option>
<option selected="selected" value="128">Medium</option>
<option value="0">Low</option>
</select>
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl03$txtManageUsersFullName" type="text" value="First2 Last2" maxlength="50" id="MainContent_grdUsers2_txtManageUsersFullName_1" style="background-color: rgb(206, 214, 231); width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%; cursor: auto;" autocomplete="off">
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl03$txtManageUsersEmail" type="text" value="user2@company.com" maxlength="50" id="MainContent_grdUsers2_txtManageUsersEmail_1" style="background-color: rgb(206, 214, 231); width: 95%; background-image: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAASCAYAAABSO15qAAAAAXNSR0IArs4c6QAAAUBJREFUOBGVVE2ORUAQLvIS4gwzEysHkHgnkMiEc4zEJXCMNwtWTmDh3UGcYoaFhZUFCzFVnu4wIaiE+vvq6+6qTgthGH6O4/jA7x1OiCAIPwj7CoLgSXDxSjEVzAt9k01CBKdWfsFf/2WNuEwc2YqigKZpK9glAlVVwTTNbQJZlnlCkiTAZnF/mePB2biRdhwHdF2HJEmgaRrwPA+qqoI4jle5/8XkXzrCFoHg+/5ICdpm13UTho7Q9/0WnsfwiL/ouHwHrJgQR8WEwVG+oXpMPaDAkdzvd7AsC8qyhCiKJjiRnCKwbRsMw9hcQ5zv9maSBeu6hjRNYRgGFuKaCNwjkjzPoSiK1d1gDDecQobOBwswzabD/D3Np7AHOIrvNpHmPI+Kc2RZBm3bcp8wuwSIot7QQ0PznoR6wYSK0Xb/AGVLcWwc7Ng3AAAAAElFTkSuQmCC"); background-repeat: no-repeat; background-attachment: scroll; background-size: 16px 18px; background-position: 98% 50%; cursor: auto;" autocomplete="off">
</td><td align="center">
<select name="ctl00$MainContent$grdUsers2$ctl03$ddlManageUsersStatus" id="MainContent_grdUsers2_ddlManageUsersStatus_1" style="background-color:#CED6E7;width:95%;">
<option selected="selected" value="Active">Active</option>
<option value="Inactive">Inactive</option>
<option value="Terminated">Terminated</option>
</select>
</td><td align="center">
<input name="ctl00$MainContent$grdUsers2$ctl03$txtManageUsersLogonTries" type="text" value="0" maxlength="1" id="MainContent_grdUsers2_txtManageUsersLogonTries_1" style="background-color:#CED6E7;width:95%;">
</td>
</tr>
</tbody>
</table>
I am trying to scrape a table that contains text, dropdown options, and values. 
我正在尝试抓取一个包含文本，下拉选项和值的表。 The result would look: user1 | 结果将如下所示：user1 | Supervisor | 主管| Medium | 中| First1 Last1 | 首页1末页1 | user1@company.com | user1@company.com | Inactive 不活跃
user2 | user2 | Supervisor | 主管| Medium | 中| First2 Last2 | First2 Last2 | user2@company.com | user2@company.com | Active 活性
Intended to be outputted to csv. 打算输出到csv。 So far I have: 到目前为止，我有：
headers = [c.get_text(strip=True) for c in soup.find('tr', attrs={'class':'listHeader'}).findAll('th')]
#find_all doesn't work here it just grabs one
for table in soup.find('table', attrs={'id':'MainContent_grdUsers2'}):
try:
column3=(table.find("option", attrs={"selected": "selected"}).get('value'))
except:
continue
#this only grabs a specific cell
for table in soup.find('table', attrs={'id':'MainContent_grdUsers2'}):
try:
column6=(table.find("input", attrs={"id": "MainContent_grdUsers2_txtManageUsersEmail_0"}).get('value'))
except:
continue
I can go in and individually grab the cells I want but there are around 100 rows of records in this table and I am finding it difficult to figure out how to grab it all at once since there isn't just text, but dropdown option values, and values. 我可以进入并单独获取所需的单元格，但是此表中大约有100行记录，而且我发现很难弄清楚如何一次获取所有记录，因为不仅有文本，而且还有下拉选项值和值。 Is there a way to do this with Beautifulsoup? 有办法用Beautifulsoup做到这一点吗？ I tried briefly with pandas and lxml but I have never used those before. 我曾短暂地尝试过使用pandas和lxml，但是以前从未使用过。
Updated code: 更新的代码：
headers = [c.get_text(strip=True) for c in soup.find('tr', attrs={'class':'listHeader'}).findAll('th')]
table = soup.find('table', attrs={'id':'MainContent_grdUsers2'})
data = []
for tr in table.find_all('tr')[1:] :
td = tr.find_all('td')
try :
data += [
[
td[0].getText() ,
td[2].find('option', {'selected':'selected'}).getText(),
td[3].find('option', {'selected':'selected'}).getText(),
td[4].find('input').get('value'),
if value is None:
continue
td[5].find('input').get('value'),
td[6].find('option', {'selected':'selected'}).getText()
]
]
except Exception as ex :
#print(ex) ## you can uncomment this line for debugging ##
continue
for row in data :
print(' '.join(row))
1 个解决方案
解决方案1
1 已采纳 2017-04-13 17:14:59
Given the html you provided, this should work : 鉴于您提供的html，这应该可以工作：
if soup.find('tr', attrs={'class':'listHeader'}) :
headers = [
'none' if c is None else c.get_text(strip=True)
for c in soup.find('tr', attrs={'class':'listHeader'}).findAll('th')
]
else :
headers = None
table = soup.find('table', attrs={'id':'MainContent_grdUsers2'})
data = []
for tr in table.find_all('tr')[1:] :
td = tr.find_all('td')
try :
data += [
[
td[0].getText() ,
td[2].find('option', {'selected':'selected'}).getText(),
td[3].find('option', {'selected':'selected'}).getText(),
td[4].find('input').get('value'),
td[5].find('input').get('value'),
td[6].find('option', {'selected':'selected'}).getText()
]
]
except Exception as ex :
#print(ex) ## you can uncomment this line for debugging ##
continue
for row in data :
print(' '.join(str(r) for r in row))
Output: 输出：
user1 Supervisor Medium First1 Last1 user1@company.com Inactive
user2 Supervisor Medium First2 Last2 user2@company.com Active
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
使用 BeautifulSoup 和 Python 抓取表格 - Scraping Table using BeautifulSoup and Python
使用 python 和 beautifulsoup 从 html 抓取一张表 - Scraping a table from html using python and beautifulsoup
在 Python 中使用 BeautifulSoup 进行表数据抓取 - Table data Scraping using BeautifulSoup in Python
python BeautifulSoup表刮 - python BeautifulSoup table scraping
使用BeautifulSoup刮表 - Scraping a table using BeautifulSoup
使用 Python BeautifulSoup 抓取表格时遇到问题 - Trouble Scraping a Table with Python BeautifulSoup
使用Python的BeautifulSoup抓取表格ID - BeautifulSoup scraping table id with python
Python 正在刮一张可扩展的桌子（BeautifulSoup）？ - Python scraping an expandable table(BeautifulSoup)?
从 Python Beautifulsoup 中抓取表格 - Scraping table from Python Beautifulsoup
Python beautifulsoup，抓取网站中的表格 - Python beautifulsoup, scraping a table in a website
相关标签