Nested lists of different lenghts from a Json to Pandas Dataframe

Question

I am having some issues with converting a JSON composed of Lists of different lenghts to a pandas dataframe. I get the JSON from a webpage like this:

import requests, json
import numpy as np

r = requests.get('https:a_web_page')

data = r.json()
type(data)

From this I got as an output that "data" is a list.

The JSON file from the webpage look like this:

[ [ 1411333200000 , 0.0 , [ 0.0 , 0.0 , 10.0 , 5.4014 , 0.42247 , 0.2517 , 
0.0 , 0.0 , 0.0 , 0.0 , 0.0616 , 0.0 , 0.0]] , 
[ 1411419600000 , 0.0 , [ 0.0 
, 0.0 , 10.0 , 4.8029 , 2.17222 , 2.216 , 0.0 , 0.0 , 0.0 , 0.0 , 0.800 , 
0.0 , 0.0]] ,
[ 1411506000000 , 1.13383 , [ 9.448 , 0.0 , 10.0 , 6.07 , 2.1722 , 2.97500 , 
0.0 , 0.0 , 0.0 , 0.0 , 3.8017 , 1.17 , 2.893]]]

Which is a correct JSON file as it can be proved in this web page: https://konklone.io/json/

What I am trying to do is convert the variable "data" to a Dataframe like this:

1411333200000   0   0   0   10  5.4014  0.42247 0.2517  0   0   0   0   
0.0616  0   0
1411419600000   0   0   0   10  4.8029  2.17222 2.216   0   0   0   0   0.8 
0   0
1411506000000   1.13383 9.448   0   10  6.07    2.1722  2.975   0   0   0   
0   3.8017  1.17    2.893

I mean, a dataframe with 15 columns and 3 rows.

With the normal way to pass a list to a DataFrame: df = pd.DataFrame(data), I get 3 columns, the 3th one is a list of 13 elements, so it doesn´t work:

1411333200000   0   [0  0   10  5.4014  0.42247 0.2517  0   0   0   0   
0.0616  0   0]
1411419600000   0   [0  0   10  4.8029  2.17222 2.216   0   0   0   0   0.8 
0   0]
1411506000000   1.13383 [9.448  0   10  6.07    2.1722  2.975   0   0   0   
0   3.8017  1.17    2.893]

I also tryed to flatten "data" with the following code, but it also didn´t work:

from collections import OrderedDict

def flatten(json_object, container=None, name=''):
    if container is None:
        container = OrderedDict()
    if isinstance(json_object, dict):
        for key in json_object:
            flatten(json_object[key], container=container, name=name + key + 
'_')
    elif isinstance(json_object, list):
        for n, item in enumerate(json_object, 1):
            flatten(item, container=container, name=name + str(n) + '_')
    else:
        container[str(name[:-1])] = str(json_object)
    return container

I also tryed unstack and json_normalize, but I coudn´t make it work. I don´t know what else to do, If any one can give me a hand or make me an advice would be nice..

Answer 1

df=pd.DataFrame(data)

df=pd.concat([df.iloc[:,0:2],df[2].apply(pd.Series)],axis=1)
df.columns=list(range(df.shape[1]))
df
Out[63]: 
              0        1      2    3     4       5        6       7    8   \
0  1411333200000  0.00000  0.000  0.0  10.0  5.4014  0.42247  0.2517  0.0   
1  1411419600000  0.00000  0.000  0.0  10.0  4.8029  2.17222  2.2160  0.0   
2  1411506000000  1.13383  9.448  0.0  10.0  6.0700  2.17220  2.9750  0.0   
    9    10   11      12    13     14  
0  0.0  0.0  0.0  0.0616  0.00  0.000  
1  0.0  0.0  0.0  0.8000  0.00  0.000  
2  0.0  0.0  0.0  3.8017  1.17  2.893