使用python搜索javascript中的字符串

Question

Following my previous question : how to fetch javascript contents in python

I tried to make another script which fetches the data from a javascript. After getting the webpage contents of course.

But, it's just not showing up the content I want. I want to find "content_id" from the javascript of the page. This is the page :- http://www.hulu.com/watch/815743

Here's what I have right now.

import re
import requests
from bs4 import BeautifulSoup
import os
import fileinput


Link = 'http://www.hulu.com/watch/815743'
q = requests.get(Link)
soup = BeautifulSoup(q.text)
#print soup
subtitles = soup.findAll('script',{'type':'text/javascript'})
pattern = re.compile(r'"content_id":"(.*?)"', re.MULTILINE | re.DOTALL)
script = soup.find("script", text=pattern)
print pattern.search(script.text).group(1)

I get this error :

AttributeError: 'NoneType' object has no attribute 'text'

Any idea how to solve this issue..?

Answer 1

There are two problems in your regular expression pattern:

• the quotes are escaped with backslashes in the script contents, take that into account
• there is a whitespace after the colon

Here is the fixed version:

pattern = re.compile(r'\\"content_id\\":\s*\\"(.*?)\\"', re.MULTILINE | re.DOTALL)

Works for me, getting 60585710 as a result.

FYI, here is the complete code that I'm executing:

import re

import requests
from bs4 import BeautifulSoup

Link = 'http://www.hulu.com/watch/815743'
q = requests.get(Link)
soup = BeautifulSoup(q.text)

pattern = re.compile(r'\\"content_id\\":\s*\\"(.*?)\\"', re.MULTILINE | re.DOTALL)
script = soup.find("script", text=pattern)
print pattern.search(script.text).group(1)