How to convert json to pandas dataframe?

Question

I am new at api programming. I am trying to download data from the moex api.

Here is the code I use:

import requests as re
from io import StringIO
import pandas as pd
import json
    
    
session = re.Session()


login = "aaaa"
password = "bbbb"

session.get('https://passport.moex.com/authenticate', auth=(login, password))    

cookies = {'MicexPassportCert': session.cookies['MicexPassportCert']}  

def api_query(engine, market, session, secur, from_start, till_end):

    param = 'https://iss.moex.com/iss/history/engines/{}/markets/{}/sessions/{}/securities/{}/candles.json?from={}&till={}&interval=24&start=0'.format(engine, market, session, secur, from_start, till_end)

    return param

url = api_query('stock', 'bonds', 'session', 'RU000A0JVWL2', '2020-11-01', '2021-05-01')

response = re.get(url, cookies=cookies)

As a result I have got the following data (part of data)

'history.cursor': {'metadata': {'INDEX': {'type': 'int64'}, 'TOTAL': {'type': 'int64'}, 'PAGESIZE': {'type': 'int64'}}, 'columns': ['INDEX', 'TOTAL', 'PAGESIZE'], 'data': [[0, 32, 100]]}}

I need to convert json format into pandas dataframe. How to do it? As a result I should get dataframe with 1 row and 3 columns.

Thanks in advance

Answer 1

Assuming your json is properly encoded you could try something like this:

import pandas as pd
import numpy as np

json = {
    'history.cursor': {
      'metadata': {'INDEX': {'type': 'int64'}, 'TOTAL': {'type': 'int64'}, 'PAGESIZE': {'type': 'int64'}}, 
      'columns': ['INDEX', 'TOTAL', 'PAGESIZE'], 
      'data': [[0, 32, 100]]
    }
}

columns = json['history.cursor']['columns']
data = np.array(json['history.cursor']['data'])
metadata = json['history.cursor']['metadata']
d = {}

for i, column in enumerate(columns):
    d[column] = data[:,i].astype(metadata[column]['type'])


df = pd.DataFrame(d)
print(df)

Answer 2

you should use the method pd.io.json.read_json() method

your orientation would likely be 'split'

so pd.read_json(json,orient='split') where split is your json in the form of dict like {index -> [index], columns -> [columns], data -> [values]}