Convert list of nested json files into pandas dataframe

Question

I have a map with a lot of json files with random names. Each file has a nested object. I want to get the files' data into a panda dataframe, with the first level being the identifiers for the nested object.

The file is as below. I have the following identifiers: seller_name, seller_location, sample_time, seller_average_response_time, fiverr_url, "seller_registration_time, gig_title. The reviews are the nested objects.

I want a dataframe that puts the identifiers for every row, and one review per row. I heard that I have to use a certain melt command for this.

Can you give an example code?

{"seller_name": "let_me_do_it_", 
"seller_location": "Austria", 
"sample_time": "21-11-2018", 
"reviews": 
[{"review_time": "about 1 year ago", 
"buyer_comment": "Good communication.", 
"buyer_name": "fivejobus", 
"buyer_feedback_rating": "5"}, 
{"review_time": "about 1 year ago", 
"buyer_comment": "Good! Thanks.", "buyer_name": "ericzhu1204",
"buyer_feedback_rating": "5"}, {"review_time": "about 1 year ago", 
"buyer_comment": "Delivery on time and Good communication,", 
"buyer_name": "fivejobus", "buyer_feedback_rating": "5"}], 
"seller_average_response_time": "", 
"fiverr_url": "https://www.fiverr.com/let_me_do_it_/translate-your-text-in-well-written-english-or-german?context&context_referrer=search_gigs&context_type=auto&pos=39&ref_ctx_id=b833b214-2869-487b-9721-fb91c0a18fb6&funnel=a316bb03-214f-44ee-a234-58e1bc3ed8e1", 
"seller_registration_time": "Aug 2017", 
"gig_title": "I will translate your english text to well written german"}

Currently, I have got this:

import os, json
import pandas as pd

path_to_json = '/Users/rogier/Downloads/data'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]
#print(json_files)  # for me this prints ['foo.json']

jsons_data = pd.DataFrame(columns=(['sellername', 'sellerlocation', 'sampletime', 'selleraverageresponsetime', 'fiverr_url', 'gigtitle'], ['review_time','buyer_comment','buyer_name','buyer_feedback_rating']))

for index, js in enumerate(json_files):
    with open(os.path.join(path_to_json, js)) as json_file:
        json_text = json.load(json_file)

        sellername = json_text['seller_name']
        sellerlocation=json_text['seller_location']
        sampletime=json_text['sample_time']

        jsons_data.loc[index] = [sellername, sellerlocation, sampletime]

I get this error:

ValueError: cannot set a row with mismatched columns

Answer 1

apply + Series

df = pd.DataFrame(my_dict)
review_data = df.reviews.apply(pd.Series)
new_df = pd.concat([df,review_data], axis = 1).drop(['reviews'], axis = 1)

Which will add each field of the dictionary as a new column of the original df :

print(df.columns)
Index(['fiverr_url', 'gig_title', 'sample_time',
   'seller_average_response_time', 'seller_location', 'seller_name',
   'seller_registration_time', 'buyer_comment', 'buyer_feedback_rating',
   'buyer_name', 'review_time'],
  dtype='object')