如何将任何嵌套的json转换为pandas数据框

Question

I'm currently working on a project that will be analyzing multiple data sources for information, other data sources are fine but I am having a lot of trouble with json and its sometimes deeply nested structure. I have tried to turn the json into a python dictionary, but with not much luck as it can start to struggle as it gets more complicated. For example with this sample json file:

{
  "Employees": [
    {
      "userId": "rirani",
      "jobTitleName": "Developer",
      "firstName": "Romin",
      "lastName": "Irani",
      "preferredFullName": "Romin Irani",
      "employeeCode": "E1",
      "region": "CA",
      "phoneNumber": "408-1234567",
      "emailAddress": "romin.k.irani@gmail.com"
    },
    {
      "userId": "nirani",
      "jobTitleName": "Developer",
      "firstName": "Neil",
      "lastName": "Irani",
      "preferredFullName": "Neil Irani",
      "employeeCode": "E2",
      "region": "CA",
      "phoneNumber": "408-1111111",
      "emailAddress": "neilrirani@gmail.com"
    }
  ]
}

after converting to dictionary and doing dict.keys() only returns "Employees". I then resorted to instead opt for a pandas dataframe and I could achieve what I wanted by calling json_normalize(dict['Employees'], sep="_") but my problem is that it must work for ALL jsons and looking at the data beforehand is not an option so my method of normalizing this way will not always work. Is there some way I could write some sort of function that would take in any json and convert it into a nice pandas dataframe? I have searched for about 2 weeks for answers bt with no luck regarding my specific problem. Thanks

Answer 1

I've had to do that in the past (Flatten out a big nested json). This blog was really helpful. Would something like this work for you?

Note, like the others have stated, for this to work for EVERY json, is a tall task, I'm merely offering a way to get started if you have a wider range of json format objects. I'm assuming they will be relatively CLOSE to what you posted as an example with hopefully similarly structures.)

jsonStr = '''{
"Employees" : [
{
"userId":"rirani",
"jobTitleName":"Developer",
"firstName":"Romin",
"lastName":"Irani",
"preferredFullName":"Romin Irani",
"employeeCode":"E1",
"region":"CA",
"phoneNumber":"408-1234567",
"emailAddress":"romin.k.irani@gmail.com"
},
{
"userId":"nirani",
"jobTitleName":"Developer",
"firstName":"Neil",
"lastName":"Irani",
"preferredFullName":"Neil Irani",
"employeeCode":"E2",
"region":"CA",
"phoneNumber":"408-1111111",
"emailAddress":"neilrirani@gmail.com"
}]
}'''

It flattens out the entire json into single rows, then you can put into a dataframe. In this case it creates 1 row with 18 columns. Then iterates through those columns, using the number values within those column names to reconstruct into multiple rows. If you had a different nested json, I'm thinking it theoretically should work, but you'll have to test it out.

import json
import pandas as pd
import re

def flatten_json(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

jsonObj = json.loads(jsonStr)
flat = flatten_json(jsonObj)



results = pd.DataFrame()
columns_list = list(flat.keys())
for item in columns_list:
    row_idx = re.findall(r'\_(\d+)\_', item )[0]
    column = item.replace('_'+row_idx+'_', '_')
    row_idx = int(row_idx)
    value = flat[item]

    results.loc[row_idx, column] = value

print (results)

Output:

print (results)
  Employees_userId           ...              Employees_emailAddress
0           rirani           ...             romin.k.irani@gmail.com
1           nirani           ...                neilrirani@gmail.com

[2 rows x 9 columns]

Answer 2

d={
"Employees" : [
{
"userId":"rirani",
"jobTitleName":"Developer",
"firstName":"Romin",
"lastName":"Irani",
"preferredFullName":"Romin Irani",
"employeeCode":"E1",
"region":"CA",
"phoneNumber":"408-1234567",
"emailAddress":"romin.k.irani@gmail.com"
},
{
"userId":"nirani",
"jobTitleName":"Developer",
"firstName":"Neil",
"lastName":"Irani",
"preferredFullName":"Neil Irani",
"employeeCode":"E2",
"region":"CA",
"phoneNumber":"408-1111111",
"emailAddress":"neilrirani@gmail.com"
}]
}
import pandas as pd
df=pd.DataFrame([x.values() for x in d["Employees"]],columns=d["Employees"][0].keys())
print(df)

Output

   userId jobTitleName firstName           ...            region  phoneNumber             emailAddress
0  rirani    Developer     Romin           ...                CA  408-1234567  romin.k.irani@gmail.com
1  nirani    Developer      Neil           ...                CA  408-1111111     neilrirani@gmail.com

[2 rows x 9 columns]

Answer 3

For the particular JSON data given. My approach, which uses pandas package only, follows:

import pandas as pd

# json as python's dict object
jsn = {
  "Employees" : [
    {
    "userId":"rirani",
    "jobTitleName":"Developer",
    "firstName":"Romin",
    "lastName":"Irani",
    "preferredFullName":"Romin Irani",
    "employeeCode":"E1",
    "region":"CA",
    "phoneNumber":"408-1234567",
    "emailAddress":"romin.k.irani@gmail.com"
    },
    {
    "userId":"nirani",
    "jobTitleName":"Developer",
    "firstName":"Neil",
    "lastName":"Irani",
    "preferredFullName":"Neil Irani",
    "employeeCode":"E2",
    "region":"CA",
    "phoneNumber":"408-1111111",
    "emailAddress":"neilrirani@gmail.com"
    }]
}

# get the main key, here 'Employees' with index '0'
emp = list(jsn.keys())[0]
# when you have several keys at this level, i.e. 'Employers' for example
# .. you need to handle all of them too (your task)

# get all the sub-keys of the main key[0] 
all_keys = jsn[emp][0].keys()

# build dataframe
result_df = pd.DataFrame()  # init a dataframe
for key in all_keys:
    col_vals = []
    for ea in jsn[emp]:
        col_vals.append(ea[key])
    # add a new column to the dataframe using sub-key as its header
    # it is possible that values here is a nested object(s)
    # .. such as dict, list, json
    result_df[key]=col_vals

print(result_df.to_string())

Output:

   userId lastName jobTitleName  phoneNumber             emailAddress employeeCode preferredFullName firstName region
0  rirani    Irani    Developer  408-1234567  romin.k.irani@gmail.com           E1       Romin Irani     Romin     CA
1  nirani    Irani    Developer  408-1111111     neilrirani@gmail.com           E2        Neil Irani      Neil     CA