python: Pandas - Convert complex json to a dataframe

Question

I have a complex JSON data structure and have to convert it to a data frame. The JSON structure is as follows:

{'fields': [{'id': 'a', 'label': 'Particulars', 'type': 'string'},
  {'id': 'b', 'label': 'States', 'type': 'string'},
  {'id': 'c', 'label': 'Gender', 'type': 'string'},
  {'id': 'd', 'label': ' 11-2013', 'type': 'string'},
  {'id': 'e', 'label': ' 12-2013', 'type': 'string'},
  {'id': 'f', 'label': ' 1-2014', 'type': 'string'},
  {'id': 'g', 'label': ' 2-2014', 'type': 'string'}],
 'data': [['Animal Husbandry- incl Poultry, Dairy and Herdsman',
   'Andhra Pradesh',
   'Men',
   '156.12',
   '153.18',
   '163.56',
   '163.56'],
  ['Animal Husbandry- incl Poultry, Dairy and Herdsman',
   'Bihar',
   'Men',
   '159.39',
   '149.38',
   '147.24',
   '155.89'],
  ['Animal Husbandry- incl Poultry, Dairy and Herdsman',
   'Gujarat',
   'Men',
   '157.08',
   '145',
   '145',
   '145']]}

I want to make a dataframe from it in the following format:

I tried directly using the read_json function which gives me error. Then I tried using json.normalize which does not give me the desired output as I don't know its proper working. Can anyone let me know how should I use json.normalize() to get the output in my required format?

Answer 1

Use json_normalize and set columns names by list comprehension:

from pandas.io.json import json_normalize

df = json_normalize(d, 'data')
df.columns = [x.get('label') for x in d['fields']]
print (df)

                                         Particulars          States Gender  \
0  Animal Husbandry- incl Poultry, Dairy and Herd...  Andhra Pradesh    Men   
1  Animal Husbandry- incl Poultry, Dairy and Herd...           Bihar    Men   
2  Animal Husbandry- incl Poultry, Dairy and Herd...         Gujarat    Men   

   11-2013  12-2013  1-2014  2-2014  
0   156.12   153.18  163.56  163.56  
1   159.39   149.38  147.24  155.89  
2   157.08      145     145     145