使用python从javascript变量JSON.parse中提取数据

Question

very new to python and trying to web scrape a website table, but I think the table data is seemingly from a Javascript variable with a JSON.parse. However the parse is not what I am used to and am unsure of how to use it in python.

The code is from this website , specifically it is var playersData = JSON.parse('\\x5B\\x7B\\x22id\\x3A,... (roughly 250,000 characters) nestled in a script tag.

So far I have managed to scrape the website using bs4, find the specific script and attempt to use re.search to find just the JSON.parse and find this <re.Match object; span=(2, 259126), match="var playersData\\t= JSON.parse('\\\\x5B\\\\x7B\\\\x22id\\> <re.Match object; span=(2, 259126), match="var playersData\\t= JSON.parse('\\\\x5B\\\\x7B\\\\x22id\\> from the search.

I would then like to export the data somewhere else after loading the JSON parse.

Here is my code so far:

import requests
from bs4 import BeautifulSoup
import json
import re

response = requests.get('https://understat.com/league/EPL/2018')
soup = BeautifulSoup(response.text, 'lxml')

playerscript = soup.find_all('script')[3].string
m = re.search("var playersData  = (.*)", playerscript)

Thanks for any help.

Answer 1

you don't need BeautifulSoup. in python json.loads same as JSON.parse and you need to convert the string using .decode('string_escape') or bytes('....', 'utf-8').decode('unicode_escape') for python 3

import requests
import json
import re

response = requests.get('https://understat.com/league/EPL/2018')
playersData = re.search("playersData\s+=\s+JSON.parse\('([^']+)", response.text)
# python 2.7
# decoded_string = playersData.groups()[0].decode('string_escape')
decoded_string = bytes(playersData.groups()[0], 'utf-8').decode('unicode_escape')
playerObj = json.loads(decoded_string)

print(playerObj[0]['player_name'])