Beautiful Soup - 提取數據僅包含 td 標簽（沒有標簽，如 div、id、class……）
[英]Beautiful Soup - Extract data only contain td tag (without tag like div, id, class…)
問題描述
我是 Beautiful Soup 的新手，我有這樣的數據，其中包含 3 組用戶數據（對於這種情況）。
我想獲取每個 USER_ID 的所有信息並保存到數據庫。
- 用戶身份
- 標題
- 內容
- PID（不是每個用戶都有這一行）
- 日期
- URL
<table align="center" border="0" style="width:550px">
<tbody>
<tr>
<td colspan="2">USER_ID 11111</td>
</tr>
<tr>
<td colspan="2">string_a</td>
</tr>
<tr>
<td colspan="2"><strong>content: aaa</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 22222</td>
</tr>
<tr>
<td colspan="2">string_b</td>
</tr>
<tr>
<td colspan="2"><strong>content: bbb</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 33333</td>
</tr>
<tr>
<td colspan="2">string_c</td>
</tr>
<tr>
<td colspan="2"><strong>content: ccc</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>PID：</strong><strong>ABCDE</strong></td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://ccc.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
</tbody>
</table>
我的問題是，
所有數據僅在 td 內，不包含 div 名稱和父標簽。 我不能分成 3 組數據。
我嘗試了以下代碼，它可以找到所有 USER_ID，但我不知道如何獲取每個 USER_ID 的其他數據
soup = BeautifulSoup(content, 'html.parser')
p = soup.find_all('td', text=re.compile("^USER_ID"))
for item in p:
title = item.find_next_siblings('td') # <--- return empty
...
我在用着
python 3.6
django 2.0.2
3 個解決方案
解決方案1
2 2020-04-30 08:32:10
from bs4 import BeautifulSoup
import re
from more_itertools import split_when
data = """<table align="center" border="0" style="width:550px">
<tbody>
<tr>
<td colspan="2">USER_ID 11111</td>
</tr>
<tr>
<td colspan="2">string_a</td>
</tr>
<tr>
<td colspan="2"><strong>content: aaa</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 22222</td>
</tr>
<tr>
<td colspan="2">string_b</td>
</tr>
<tr>
<td colspan="2"><strong>content: bbb</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 33333</td>
</tr>
<tr>
<td colspan="2">string_c</td>
</tr>
<tr>
<td colspan="2"><strong>content: ccc</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>PID：</strong><strong>ABCDE</strong></td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://ccc.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
</tbody>
</table>"""
soup = BeautifulSoup(data, 'html.parser')
target = soup.find("table", align="center")
goal = [item.text for item in target.select(
"td", text=re.compile("^USER_ID")) if item.text.strip() != '']
final = list(split_when(goal, lambda _, y: y.startswith("USER")))
print(final) # list of lists
for x in final: # or loop
print(x)
Output
[['USER_ID 11111', 'string_a', 'content: aaa', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com'], ['USER_ID 22222', 'string_b', 'content: bbb', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com'], ['USER_ID 33333', 'string_c', 'content: ccc', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'PID：ABCDE', 'URL：https://ccc.com']]
和
['USER_ID 11111', 'string_a', 'content: aaa', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com']
['USER_ID 22222', 'string_b', 'content: bbb', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com']
['USER_ID 33333', 'string_c', 'content: ccc', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'PID：ABCDE', 'URL：https://ccc.com']
解決方案2
1 2020-04-30 11:12:49
嘗試使用以下代碼來識別find_all_next('td')並檢查 if 條件以破壞dataset 。
import re
from bs4 import BeautifulSoup
html='''<table align="center" border="0" style="width:550px">
<tbody>
<tr>
<td colspan="2">USER_ID 11111</td>
</tr>
<tr>
<td colspan="2">string_a</td>
</tr>
<tr>
<td colspan="2"><strong>content: aaa</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 22222</td>
</tr>
<tr>
<td colspan="2">string_b</td>
</tr>
<tr>
<td colspan="2"><strong>content: bbb</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://aaa.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2">USER_ID 33333</td>
</tr>
<tr>
<td colspan="2">string_c</td>
</tr>
<tr>
<td colspan="2"><strong>content: ccc</strong></td>
</tr>
<tr>
<td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td>
</tr>
<tr>
<td colspan="2"><strong>PID：</strong><strong>ABCDE</strong></td>
</tr>
<tr>
<td colspan="2"><strong>URL：https://ccc.com</strong></td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
<tr>
<td colspan="2"> </td>
</tr>
</tbody>
</table>'''
soup=BeautifulSoup(html,'html.parser')
final_list=[]
for item in soup.find_all('td',text=re.compile("USER_ID")):
row_list=[]
row_list.append(item.text.strip())
siblings=item.find_all_next('td')
for sibling in siblings:
if "USER_ID" in sibling.text:
break
else:
if sibling.text.strip()!='':
row_list.append(sibling.text.strip())
final_list.append(row_list)
print(final_list)
Output ：
[['USER_ID 11111', 'string_a', 'content: aaa', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com'], ['USER_ID 22222', 'string_b', 'content: bbb', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com'], ['USER_ID 33333', 'string_c', 'content: ccc', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'PID：ABCDE', 'URL：https://ccc.com']]
如果您希望打印每個列表，請嘗試此操作。
soup=BeautifulSoup(html,'html.parser')
for item in soup.find_all('td',text=re.compile("USER_ID")):
row_list=[]
row_list.append(item.text.strip())
siblings=item.find_all_next('td')
for sibling in siblings:
if "USER_ID" in sibling.text:
break
else:
if sibling.text.strip()!='':
row_list.append(sibling.text.strip())
print(row_list)
Output ：
['USER_ID 11111', 'string_a', 'content: aaa', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com']
['USER_ID 22222', 'string_b', 'content: bbb', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'URL：https://aaa.com']
['USER_ID 33333', 'string_c', 'content: ccc', 'date：2020-05-01 00:00:00 To 2020-05-03 23:59:59', 'PID：ABCDE', 'URL：https://ccc.com']
解決方案3
-1 2020-04-30 07:55:59
你可以簡單地使用soup.select('table tr')
例子
from bs4 import BeautifulSoup
html = '<table align="center" border="0" style="width:550px"><tbody>' \
'<tr><td colspan="2">USER_ID 11111</td></tr>' \
'<tr><td colspan="2">string_a</td></tr>' \
'<tr><td colspan="2"><strong>content: aaa</strong></td></tr>' \
'<tr><td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td></tr>' \
'<tr><td colspan="2"><strong>URL：https://aaa.com</strong></td></tr>' \
'<tr><td colspan="2"> </td></tr>' \
'<tr><td colspan="2"> </td></tr>' \
'<tr><td colspan="2">USER_ID 22222</td></tr>' \
'<tr><td colspan="2">string_b</td></tr>' \
'<tr><td colspan="2"><strong>content: bbb</strong></td></tr>' \
'<tr><td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td></tr>' \
'<tr><td colspan="2"><strong>URL：https://aaa.com</strong></td></tr>' \
'<tr><td colspan="2"> </td></tr>' \
'<tr><td colspan="2"> </td></tr>' \
'<tr><td colspan="2">USER_ID 33333</td></tr>' \
'<tr><td colspan="2">string_c</td></tr>' \
'<tr><td colspan="2"><strong>content: ccc</strong></td></tr>' \
'<tr><td colspan="2"><strong>date：</strong>2020-05-01 00:00:00 To 2020-05-03 23:59:59</td></tr>' \
'<tr><td colspan="2"><strong>PID：</strong><strong>ABCDE</strong></td></tr>' \
'<tr><td colspan="2"><strong>URL：https://ccc.com</strong></td></tr>' \
'<tr><td colspan="2"> </td></tr>' \
'<tr><td colspan="2"> </td></tr></tbody></table>'
soup = BeautifulSoup(html, features="lxml")
elements = soup.select('table tr')
print(elements)
for element in elements:
print(element.text)
打印出來
USER_ID 11111
string_a
content: aaa
date：2020-05-01 00:00:00 To 2020-05-03 23:59:59
URL：https://aaa.com
USER_ID 22222
string_b
content: bbb
date：2020-05-01 00:00:00 To 2020-05-03 23:59:59
URL：https://aaa.com
USER_ID 33333
string_c
content: ccc
date：2020-05-01 00:00:00 To 2020-05-03 23:59:59
PID：ABCDE
URL：https://ccc.com
Python & Beautiful Soup - 在特定標簽和類組合之間提取文本
[英]Python & Beautiful Soup - Extract text between a specific tag and class combination
Beautiful Soup 4的extract（）將標簽更改為NoneType
[英]Beautiful Soup 4's extract() Changing the tag to a NoneType
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.
相關問題
使用美麗的湯 - 在 a 中提取字符串<div>標簽？
使用美麗的湯隔離'td'標簽
使用美麗的湯 python 從標簽內部提取 html ID
表td的Python美麗湯標簽
Python Beautiful Soup打印確切的TD標簽
Python & Beautiful Soup - 在特定標簽和類組合之間提取文本
Beautiful Soup - 在沒有嵌套跨度的情況下獲得頂部 div 標簽
Python Beautiful Soup在div標簽內提取數據
Beautiful Soup 4的extract（）將標簽更改為NoneType
用美麗的湯從html標簽中提取字符串
相關標簽