使用Python从网站图表中收集数据

Question

As an exercise, I am trying to scrape data from a dynamic graph using Python. The graph can be found at this link (let's say I want the data from the first one).

Now, I was thinking of doing something like:

src = 'https://marketchameleon.com/Overview/WFT/IV/#_ABSTRACT_RENDERER_ID_11'

import json
import urllib.request

with urllib.request.urlopen(src) as url:
    data = url.read()
    reply = json.loads(data)

However, I receive an error message on the last line of the code, saying:

JSONDecodeError: Expecting value

"data" is not empty, so I believe there is a problem with the format of the information within it. Does someone have an idea to solve this issue? Thanks!

Answer 1

I opened that link and see that the site loads data from another URL - https://marketchameleon.com/charts/histStockChartData?p=747&m=12&_=1534060722519

You can use json.loads() function twice and do some hacks with headers ( urllib2.Request is your friend in case of Python 2) since server returns HTTP 500 when you don't imitate browser

src = 'https://marketchameleon.com/charts/histStockChartData?p=747&m=12'

import json
import urllib.request

user_agent = {
    'Host': 'marketchameleon.com',
    'Connection': 'keep-alive',
    'Pragma': 'no-cache',
    'Cache-Control': 'no-cache',
    'Upgrade-Insecure-Requests': 1,
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Language': 'ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,kk;q=0.6'
}
request = urllib.request.Request(src, headers=user_agent)

data = urllib.request.urlopen(request).read()
print(data)
reply = json.loads(data)

table = json.loads(reply['GTable'])
print(table)