Python：使用BeatifulSoup从数据绑定中获取数据

Question

This is an expand from a question that I posted a week ago ( getting text from html using beatifulsoup ). It seems that most of the data that I want to extract is data-bind and is not 'stored' when i use soup.findAll . For example taking this link: kaggle/user/results I am trying to get the name of all the competitions the user participated. I am using the following code:

url = 'https://www.kaggle.com/titericz/results'
sourceCode = requests.get(url)
plainText = sourceCode.text
soup = BeautifulSoup(plainText)
for link in soup.findAll('tr'):
    print(link)

So i take the first competition but in the link it seems that the values of name of competition, position in this competition, total competitors etc. are missing while in the html are there. Tried to follow the same procedure with the answer of the question that I link above, but I could not manage it(by using re.compile and pattern.search ). Is there a way to accomplish it by using BeatifulSoup ? I couldnt find any similar issue on the web.

Answer 1

You can parse the underlying get request, which returns a json string.

here's a small script which will get you started.

import requests
import json

jsonResponse = requests.get("https://www.kaggle.com/knockout/profiles/54836/results")
data = json.loads(jsonResponse.text)
print(data)

for eachData in data:
    print("competition name:", eachData["competition"]["title"])
    print("Rank:", eachData["rank"])
    print("competitors count:", eachData["teamCount"])

the output will be of the format:

 competition name: Digit Recognizer 
 Rank: None
 competitors count: 933
 competition name: The Allen AI Science Challenge 
 Rank: 110 
 competitors count: 486