刮下桌子的底行
[英]Scraping for bottom row of table
問題描述
我正在使用python 3.4。 我知道如何利用BeautifulSoup來抓取網頁，但我正在努力想出最有效的方法來實現這一目標。 Nexus工廠映像頁面 （Android）包含所有Nexus設備的列表，並在新版本可用時更新。 最新版本始終添加到相應表格的底部。 我有一個每個設備的名稱列表，包括真實姓名和代號，我只提取這些（設備本身只更新一次/年，如果那樣，只有部分設備仍然接收更新）。
從每個表中拉出底部條目的最有效方法是什么？ 我計划將底行中第一個<td>中的每個字符串保存為pickle對象，這樣我以后可以輕松比較字符串以檢查當前底行是否是新的，但我不確定刮擦的最佳方法是什么對於入口本身。
每個<tr>都有一個devnamebuildnumber格式的id。 因為我有每個設備的名稱並且將具有最新的字符串，所以我應該能夠使用soup.find("tr", id=dev + buildstring) 。 然而，這會返回找到的行的每個兄弟和孩子，所以我不確定如何最好地利用它。
2 個解決方案
解決方案1
2 已采納 2016-01-25 18:11:52
這是讓你入門的東西。 我們的想法是獲取具有id屬性的h2元素 - 除了第一個元素，這些元素將是設備名稱元素。 對於找到的每個元素，讓我們獲取下一個table元素並將版本解析為列表。 執行：
from pprint import pprint
import requests
from bs4 import BeautifulSoup
url = "https://developers.google.com/android/nexus/images"
response = requests.get(url)
soup = BeautifulSoup(response.content, "lxml")
data = {}
for device in soup.find_all("h2", id=True)[1:]:
device_name = device.get_text(strip=True)
data[device_name] = [version.find("td").get_text(strip=True)
for version in device.find_next("table").find_all("tr", id=True)]
pprint(data)
使用設備名稱作為鍵打印字典，並將版本作為值打印：
{'"angler" for Nexus 6P': ['6.0.0 (MDA89D)',
'6.0.0 (MDB08K)',
'6.0.0 (MDB08L)',
'6.0.0 (MDB08M)',
'6.0.0 (MMB29N)',
'6.0.1 (MMB29M)',
'6.0.1 (MMB29P)'],
'"bullhead" for Nexus 5X': ['6.0.0 (MDA89E)',
'6.0.0 (MDB08I)',
'6.0.0 (MDB08L)',
'6.0.0 (MDB08M)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29P)'],
'"fugu" for Nexus Player': ['5.0 (LRX21M)',
'5.0 (LRX21V)',
'5.1.0 (LMY47D)',
'5.1.1 (LMY47V)',
'5.1.1 (LMY48J)',
'5.1.1 (LMY48N)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.1 (MMB29M)',
'6.0.1 (MMB29T)'],
'"hammerhead" for Nexus 5 (GSM/LTE)': ['4.4 (KRT16M)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84M)',
'4.4.4 (KTU84P)',
'4.4.4 Release 2 (For 2Degrees/NZ, '
'Telstra/AUS and India ONLY) (KTU84Q)',
'5.0 (LRX21O)',
'5.0.1 (LRX22C)',
'5.1.0 (LMY47D)',
'5.1.0 (LMY47I)',
'5.1.1 (LMY48B)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29S)'],
'"mantaray" for Nexus 10': ['4.2.2 (JDQ39)',
'4.3 (JWR66Y)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0 (LRX21P)',
'5.0.1 (LRX22C)',
'5.0.2 (LRX22G)',
'5.1.0 (LMY47D)',
'5.1.1 (LMY47V)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'5.1.1 (LMY48T)',
'5.1.1 (LMY48X)',
'5.1.1 (LMY48Z)',
'5.1.1 (LMY49F)'],
'"mysid" for Galaxy Nexus "toro" (Verizon CDMA/LTE)': ['4.0.4 (IMM76K)',
'4.1.1 (JRO03O)',
'4.2.2 (JDQ39)'],
'"mysidspr" for Galaxy Nexus "toroplus" (Sprint CDMA/LTE)': ['4.1.1 (FH05)',
'4.2.1 (GA02)'],
'"nakasi" for Nexus 7 (Wi-Fi)': ['4.1.2 (JZO54K)',
'4.2.2 (JDQ39)',
'4.3 (JWR66Y)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0 (LRX21P)',
'5.0.2 (LRX22G)',
'5.1.0 (LMY47D)',
'5.1.1 (LMY47V)'],
'"nakasig" for Nexus 7 (Mobile)': ['4.2.2 (JDQ39)',
'4.3 (JWR66Y)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0.2 (LRX22G)',
'5.1.0 (LMY47D)',
'5.1.1 (LMY47V)'],
'"occam" for Nexus 4': ['4.2.2 (JDQ39)',
'4.3 (JWR66Y)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0 (LRX21T)',
'5.0.1 (LRX22C)',
'5.1.0 (LMY47O)',
'5.1.1 (LMY47V)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'5.1.1 (LMY48T)'],
'"razor" for Nexus 7 [2013] (Wi-Fi)': ['4.3 (JSS15Q)',
'4.3 (JSS15R)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0 (LRX21P)',
'5.0.1 (LRX22C)',
'5.0.2 (LRX22G)',
'5.1.0 (LMY47O)',
'5.1.1 (LMY47V)',
'5.1.1 (LMY48G)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'5.1.1 (LMY48T)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58U)',
'6.0.0 (MRA58V)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29O)'],
'"razorg" for Nexus 7 [2013] (Mobile)': ['4.3 (JLS36C)',
'4.3.1 (JLS36I)',
'4.4 (KRT16S)',
'4.4.2 (KOT49H)',
'4.4.2_r2 (Verizon) (KVT49L)',
'4.4.3 (KTU84L)',
'4.4.4 (KTU84P)',
'5.0.2 (LRX22G)',
'5.1.0 (LMY47O)',
'5.1.1 (LMY47V)',
'5.1.1 (LMY48P)',
'5.1.1 (LMY48U)',
'5.1.1 (LMY48X)',
'5.1.1 (LMY48Z)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.0 (MRA58V)',
'6.0.0 (MRA59B)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29O)'],
'"ryu" for Pixel C': ['6.0.1 (MXB48J)', '6.0.1 (MXB48K)'],
'"shamu" for Nexus 6': ['5.0 (LRX21O)',
'5.0.1 (LRX22C)',
'5.1.0 (LMY47D)',
'5.1.0 (LMY47E)',
'5.1.0 (LMY47I)',
'5.1.0 (For T-Mobile ONLY) (LMY47M)',
'5.1.1 (All carriers except T-Mobile US) (LMY47Z)',
'5.1.1 (For T-Mobile ONLY) (LYZ28E)',
'5.1.1 (For Project Fi ONLY) (LVY48C)',
'5.1.1 (LMY48I)',
'5.1.1 (For T-Mobile ONLY) (LYZ28J)',
'5.1.1 (For Project Fi ONLY) (LVY48E)',
'5.1.1 (LMY48M)',
'5.1.1 (For T-Mobile ONLY) (LYZ28K)',
'5.1.1 (For Project Fi ONLY) (LVY48F)',
'5.1.1 (LMY48T)',
'5.1.1 (For T-Mobile ONLY) (LYZ28M)',
'5.1.1 (For Project Fi ONLY) (LVY48H)',
'5.1.1 (LMY48W)',
'5.1.1 (LMY48X)',
'5.1.1 (LMY48Y)',
'5.1.1 (For T-Mobile ONLY) (LYZ28N)',
'5.1.1 (For Project Fi ONLY) (LVY48I)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.0 (MRA58R)',
'6.0.0 (MRA58X)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29S)'],
'"soju" for Nexus S (worldwide version, i9020t and i9023)': ['2.3.6 (GRK39F)',
'4.0.4 (IMM76D)',
'4.1.2 (JZO54K)'],
'"sojua" for Nexus S (850MHz version, i9020a)': ['2.3.6 (GRK39F)',
'4.0.4 (IMM76D)',
'4.1.2 (JZO54K)'],
'"sojuk" for Nexus S (Korea version, m200)': ['2.3.6 (GRK39F)',
'4.0.4 (IMM76D)',
'4.1.1 (JRO03E)'],
'"sojus" for Nexus S 4G (d720)': ['2.3.7 (GWK74)',
'4.0.4 (IMM76D)',
'4.1.1 (JRO03R)'],
'"takju" for Galaxy Nexus "maguro" (GSM/HSPA+) (with Google Wallet)': ['4.0.4 '
'(IMM76I)',
'4.1.2 '
'(JZO54K)',
'4.2.2 '
'(JDQ39)',
'4.3 '
'(JWR66Y)'],
'"tungsten" for Nexus Q': ['4.0.4 (IAN67K)'],
'"volantis" for Nexus 9 (Wi-Fi)': ['5.0 (LRX21Q)',
'5.0 (LRX21R)',
'5.0.1 (LRX22C)',
'5.0.2 (LRX22L)',
'5.1.1 (LMY47X)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'5.1.1 (LMY48T)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29S)'],
'"volantisg" for Nexus 9 (LTE)': ['5.0.1 (LRX22C)',
'5.0.2 (LRX22L)',
'5.1.1 (LMY47X)',
'5.1.1 (LMY48I)',
'5.1.1 (LMY48M)',
'5.1.1 (LMY48T)',
'5.1.1 (LMY48X)',
'5.1.1 (LMY48Z)',
'5.1.1 (LMY49F)',
'6.0.0 (MRA58K)',
'6.0.0 (MRA58N)',
'6.0.1 (MMB29K)',
'6.0.1 (MMB29S)'],
'"yakju" for Galaxy Nexus "maguro" (GSM/HSPA+)': ['4.0.4 (IMM76I)',
'4.1.2 (JZO54K)',
'4.2.2 (JDQ39)',
'4.3 (JWR66Y)']}
解決方案2
1 2016-01-25 18:32:35
以下內容生成一個列表，其中包含每個設備的最后一個條目 要做到這一點，你仍然需要遍歷所有項目，但是只需保留最后一個條目，如下所示：
from bs4 import BeautifulSoup
import requests
html = requests.get("https://developers.google.com/android/nexus/images")
soup = BeautifulSoup(html.text, "lxml")
models = []
for h2 in soup.find_all('h2', id=True)[1:]:
tr = h2.find_next('table').find_all('tr', id=True)[-1]
td = [t.text.strip() for t in tr.find_all('td')]
models.append([h2.text] + td)
for device, version, link, cs1, cs2 in models:
print '{}, {}'.format(device, version)
這顯示以下內容：
"ryu" for Pixel C, 6.0.1 (MXB48K)
"angler" for Nexus 6P, 6.0.1 (MMB29P)
"bullhead" for Nexus 5X, 6.0.1 (MMB29P)
"shamu" for Nexus 6, 6.0.1 (MMB29S)
"fugu" for Nexus Player, 6.0.1 (MMB29T)
"volantisg" for Nexus 9 (LTE), 6.0.1 (MMB29S)
"volantis" for Nexus 9 (Wi-Fi), 6.0.1 (MMB29S)
"hammerhead" for Nexus 5 (GSM/LTE), 6.0.1 (MMB29S)
"razor" for Nexus 7 [2013] (Wi-Fi), 6.0.1 (MMB29O)
"razorg" for Nexus 7 [2013] (Mobile), 6.0.1 (MMB29O)
"mantaray" for Nexus 10, 5.1.1 (LMY49F)
"occam" for Nexus 4, 5.1.1 (LMY48T)
"nakasi" for Nexus 7 (Wi-Fi), 5.1.1 (LMY47V)
"nakasig" for Nexus 7 (Mobile), 5.1.1 (LMY47V)
"tungsten" for Nexus Q, 4.0.4 (IAN67K)
"takju" for Galaxy Nexus "maguro" (GSM/HSPA+) (with Google Wallet), 4.3 (JWR66Y)
"yakju" for Galaxy Nexus "maguro" (GSM/HSPA+), 4.3 (JWR66Y)
"mysid" for Galaxy Nexus "toro" (Verizon CDMA/LTE), 4.2.2 (JDQ39)
"mysidspr" for Galaxy Nexus "toroplus" (Sprint CDMA/LTE), 4.2.1 (GA02)
"soju" for Nexus S (worldwide version, i9020t and i9023), 4.1.2 (JZO54K)
"sojua" for Nexus S (850MHz version, i9020a), 4.1.2 (JZO54K)
"sojuk" for Nexus S (Korea version, m200), 4.1.1 (JRO03E)
"sojus" for Nexus S 4G (d720), 4.1.1 (JRO03R)
在python中使用Selenium進行抓取時，只能獲得表中的第一行
[英]Can only get first row in table when scraping with Selenium in python
抓取和解析 BeautifulSoup TR:TD 表時空白初始行
[英]Blank initial row when scraping and parsing BeautifulSoup TR:TD table
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.