尝试使用Python从URL获取json数据

Question

I am learning to get json data from a link and use that data later on. But i am getting error: "RuntimeError: maximum recursion depth exceeded while calling a Python object"

Here is my code:

import json
import requests
from bs4 import BeautifulSoup

url = "http://example.com/category/page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}

mainPage = requests.get(url, headers = header)
xTree = BeautifulSoup(mainPage.content, "lxml")

newDictionary=json.loads(str(xTree))

print (newDictionary)

EDIT: Okay I got the response data from using this slight change, here is the new code:

import json
import requests
from bs4 import BeautifulSoup

url = "http://example.com/category/page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}

mainPage = requests.get(url, headers = header

print (mainPage.json())

Answer 1

Don't use beautiful soup to process a json http response. Use something like requests:

url = "https://www.daraz.pk/womens-kurtas-shalwar-kameez/?pathInfo=womens-kurtas-shalwar-kameez&page=2&YII_CSRF_TOKEN=31eb0a5d28f4dde909d3233b5a0c23bd03348f69&more_products=true"
header = {'x-requested-with': 'XMLHttpRequest'}
t = requests.get(url, headers=True)
newDictionary=json.loads(t)
print (newDictionary)

The beautiful soup object can't be parsed with json.loads() that way.

If you have HTML data on some of those json keys then you can use beautiful soup to parse those string values individually. If you have a key called content on your json, containing html, you can parse it like so:

BeautifulSoup(newDictionary.content, "lxml")

You may need to experiment with different parsers, if you have fragmentary html.

Answer 2

The following is an example of how to use various JSON data that has been loaded as an object with json.loads() .

Working Example — Tested with Python 2.6.9 and 2.7.10 and 3.3.5 and 3.5.0

import json

json_data = '''
{
    "array": [
        1,
        2,
        3
    ],
    "boolean": true,
    "null": null,
    "number": 123,
    "object": {
        "a": "b",
        "c": "d",
        "e": "f"
    },
    "string": "Hello World"
}
'''

data = json.loads(json_data)

list_0 = [
    data['array'][0],
    data['array'][1],
    data['array'][2],
    data['boolean'],
    data['null'],
    data['number'],
    data['object']['a'],
    data['object']['c'],
    data['object']['e'],
    data['string']
]

print('''
array value 0           {0}
array value 1           {1}
array value 2           {2}
boolean value           {3}
null value              {4}
number value            {5}
object value a value    {6}
object value c value    {7}
object value e value    {8}
string value            {9}
'''.format(*list_0))

Output

array value 0           1
array value 1           2
array value 2           3
boolean value           True
null value              None
number value            123
object value a value    b
object value c value    d
object value e value    f
string value            Hello World