I am scraping a page and cannot get a certain field because it is stored in a javascript variable.
My question is, how can I scrape the following code, decode it, and save the <li>
tag(s) content? Using BeautifulSoup and anything else python.
Here is the code inside the <script>
tag:
<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>
It is horribly formatted, I apologize.
I need to save the urls inside of all the li
tags only for the variable html_role_listing
.
\u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E
/casting/untitled-comedy-short-41680/martinique-159296/
If you want to do it in a generic way, you'll need a library that parses javascript. In this example, I'll use slimit
to do that.
First, load in your data:
from bs4 import BeautifulSoup as Soup
import slimit
from slimit.parser import Parser
from slimit.visitors import nodevisitor
a = """<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>"""
soup = Soup(a)
js_content = soup.findAll('script')[0].text
par = Parser()
tree = par.parse(js_content)
encoded_html = [n.value for n in nodevisitor.visit(tree) if isinstance(n, slimit.ast.String)][2]
Finally, you can easily decode that string, which is valid HTML:
decoded_html = encoded_html.decode('unicode_escape')
print(decoded_html)
role_listing = Soup(decoded_html)
output = [ anchor.attrs['href'] for anchor in role_listing.select('li a') ]
print('---')
print("\n".join(output))
'<div class="text callListing loggedout "> <p class="title"><a name="roles"></a>Seeking Talent <span class="optional">Select a role below for more information and submission instructions.</span></p> <div class="castingRoles"> <ul> <li > <a href="/casting/untitled-comedy-short-41680/martinique-159296/"> Martinique (Lead): <span class="roletag"> Female, 18-25, Caucasian </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> native French speaker. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/justin-159297/"> Justin (Lead): <span class="roletag"> Male, 20-25, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> comedy and improv skills, hopeless romantic. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/"> Flower Shop Salesperson : <span class="roletag"> Males & Females, 30+, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> impatient. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/models-159300/"> Models (Supporting): <span class="roletag"> Female, 18-35, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> small roles, under five lines. </p> </li> </ul> </div></div>'
---
/casting/untitled-comedy-short-41680/martinique-159296/
/casting/untitled-comedy-short-41680/justin-159297/
/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/
/casting/untitled-comedy-short-41680/models-159300/
No JS parser necessary!
I'm assuming you know your <script>
contents are going to be formatted as in your example, and that you've already scraped the contents of that tag into a variable called script_text
.
First, we need to get the value of html_role_listing
, which we can do with a good ol' regular expression:
>>> import re
>>> html_role_listing_match = re.search(r'var html_role_listing = \'(.+)\';$', script_text, re.MULTILINE)
>>> html_role_listing = html_role_listing_match.group(1)
Then, we take advantage of the fact that the \<
and similar escape sequences are also valid in Python Unicode strings (just as they're valid in JS strings), and parse them using the safer version of eval
:
>>> import ast
>>> roles_html = ast.literal_eval("u'%s'" % html_role_listing)
Just to prove it to yourself, you can check the first few characters of this and see that they've been properly parsed:
>>> print roles_html[:10]
<div class
Now, you have an HTML string you can parse with BeautifulSoup:
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(roles_html)
And grab those links' href
attributes.
>>> links = soup.select('li a')
>>> for link in links:
... print link.attrs['href']
...
/casting/untitled-comedy-short-41680/martinique-159296/
/casting/untitled-comedy-short-41680/justin-159297/
/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/
/casting/untitled-comedy-short-41680/models-159300/
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.