![](/img/trans.png)
[英]Cannot scrape with beautifulsoup and urllib because of javascript variable
[英]BeautifulSoup scrape from javascript (encoded) variable
我正在抓取頁面,但無法獲取某個字段,因為它存儲在javascript變量中。
我的問題是,我如何抓取以下代碼,對其進行解碼並保存<li>
標簽的內容? 使用BeautifulSoup和其他任何python。
這是<script>
標記內的代碼:
<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>
我只需要將所有li
標簽內的網址保存為變量html_role_listing
。
\u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E
/casting/untitled-comedy-short-41680/martinique-159296/
如果要以通用方式執行此操作,則需要一個解析javascript的庫。 在此示例中,我將使用slimit
來做到這一點。
首先,加載您的數據:
from bs4 import BeautifulSoup as Soup
import slimit
from slimit.parser import Parser
from slimit.visitors import nodevisitor
a = """<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>"""
soup = Soup(a)
js_content = soup.findAll('script')[0].text
現在您可以選擇:使用正則表達式解析js_content
或使用詞法分析器(在這種情況下為slimit)。
par = Parser()
tree = par.parse(js_content)
encoded_html = [n.value for n in nodevisitor.visit(tree) if isinstance(n, slimit.ast.String)][2]
decoded_html = encoded_html.decode('unicode_escape')
print(decoded_html)
role_listing = Soup(decoded_html)
output = [ anchor.attrs['href'] for anchor in role_listing.select('li a') ]
print('---')
print("\n".join(output))
'<div class="text callListing loggedout "> <p class="title"><a name="roles"></a>Seeking Talent <span class="optional">Select a role below for more information and submission instructions.</span></p> <div class="castingRoles"> <ul> <li > <a href="/casting/untitled-comedy-short-41680/martinique-159296/"> Martinique (Lead): <span class="roletag"> Female, 18-25, Caucasian </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> native French speaker. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/justin-159297/"> Justin (Lead): <span class="roletag"> Male, 20-25, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> comedy and improv skills, hopeless romantic. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/"> Flower Shop Salesperson : <span class="roletag"> Males & Females, 30+, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> impatient. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/models-159300/"> Models (Supporting): <span class="roletag"> Female, 18-35, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> small roles, under five lines. </p> </li> </ul> </div></div>'
---
/casting/untitled-comedy-short-41680/martinique-159296/
/casting/untitled-comedy-short-41680/justin-159297/
/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/
/casting/untitled-comedy-short-41680/models-159300/
無需JS解析器!
我假設您知道您的<script>
內容將像示例中一樣進行格式化,並且您已經將該標記的內容script_text
到了一個名為script_text
的變量中。
首先,我們需要獲取html_role_listing
的值,我們可以使用一個好的ol'正則表達式來做到這一點:
>>> import re
>>> html_role_listing_match = re.search(r'var html_role_listing = \'(.+)\';$', script_text, re.MULTILINE)
>>> html_role_listing = html_role_listing_match.group(1)
然后,我們利用\<
和類似的轉義序列在Python Unicode字符串中同樣有效(就像它們在JS字符串中一樣)的事實,並使用更安全的eval
版本對其進行解析:
>>> import ast
>>> roles_html = ast.literal_eval("u'%s'" % html_role_listing)
只是為了向自己證明這一點,您可以檢查其中的前幾個字符並查看它們是否已正確解析:
>>> print roles_html[:10]
<div class
現在,您有了可以用BeautifulSoup解析的HTML字符串:
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(roles_html)
並抓住這些鏈接的href
屬性。
>>> links = soup.select('li a')
>>> for link in links:
... print link.attrs['href']
...
/casting/untitled-comedy-short-41680/martinique-159296/
/casting/untitled-comedy-short-41680/justin-159297/
/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/
/casting/untitled-comedy-short-41680/models-159300/
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.