BeautifulSoup 从中提取数据<script> with unicode

Question

I am trying to parse a JSON in a script of an html page.

import requests
from bs4 import BeautifulSoup
import json

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36 OPR/64.0.3417.47",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "fr-FR,en;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "keep-alive",
    "Cache-Control": "max-age=0",
    "Upgrade-Insecure-Requests": "1",
}

proxy = "http://stack:overflow@45.135.149.142:14758"

url = "https://www.seloger.com/list.htm?projects=2%2C5&types=2%2C1&natures=1%2C2%2C4&places=%5B%7Bci%3A60088%7D%5D&enterprise=0&qsVersion=1.0"
r = requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers)
soup = BeautifulSoup(r.text, 'html.parser')
for script in soup.find_all('script'):
    if "initialData" in script.text:
        data = script.text.split('JSON.parse("', 1)[1].split('");window["tags"]', 1)[0]
        json_data = json.loads(data)

And this error is returned:

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

The problem is that " is not converted to a quotation mark " which triggers the json decoder error.

Also, whenever I print the code of the script, " is printed instead of ". I have already tried encoding and decoding in multiple formats before passing it to json.loads but nothing worked.

Th problem only comes when the code is parsed directly from the request response. I cannot replicate the issue. This code works as expected:

snippet = '''<script>
window["initialData"] = JSON.parse("{\u0022foo\u0022:\u0022bar\u0022,\u0022xxx\u0022:\u0022xyz\u0022}")
</script>
'''
soup = BeautifulSoup(snippet, 'html.parser')
for script in soup.find_all('script'):
    data = script.text.split('JSON.parse("')[1].split('")')[0]
    json_data = json.loads(data)
    print(json_data)
    # output : {'foo': 'bar', 'xxx': 'xyz'}

How can I fix this?

Answer 1

You can do it with a regex (not the best way), but this worked for me:

import requests,json,re
proxy='http://stack:overflow@45.135.149.142:14758'
url='https://www.seloger.com/list.htm?projects=2%2C5&types=2%2C1&natures=1%2C2%2C4&places=%5B%7Bci%3A60088%7D%5D&enterprise=0&qsVersion=1.0'
r = requests.get(url, proxies={"http": proxy, "https": proxy})
json_data = json.loads(json.loads('"' + re.search(r'JSON\.parse\("(.+)"\);w', r.text).group(1) + '"')) # note it needs to be double wrapped
json_data.keys()
# dict_keys(['cards', 'navigation', 'SEO', 'tracking', 'adverts', 'bookmarks', 'failure', 'engine'])

Answer 2

The content you're trying to scrape includes unicode escape sequences which seem to actually be escaped themselves.

The solution I've found involves encoding and then decoding the string, although there might be a better way:

data.encode("utf-8").decode("unicode-escape")

I also made some other tweaks to your code, particularly in how the data is parsed, seen in this snippet which I used as a demo/test:

json_re = re.compile(r"window\[\"initialData\"] = JSON\.parse\(\"(.*)\"\);window\[\"tags\"]")

with open("../out/temp.html", 'rb') as file_in:
    soup = BeautifulSoup(file_in.read(), 'lxml')

raw_data = ""
for script in soup.find_all('script'):
    if "initialData" in script.text:
        res_text = script.get_text(strip=True)
        raw_data = json_re.search(res_text).group(1)
        break

print(raw_data)
t_1 = raw_data.encode("utf-8")
print(t_1)

t_2 = t_1.decode("unicode-escape")
print(t_2)

t_3 = json.loads(t_2)
print(t_3)

Answer 3

Maybe the problem is the splitting step, but I had similar problems with json and encodings. Try to recode the response.text with 'utf-8' or 'latin'. Some times it worked for me. Hope it helps!