Python csv to json using pandas - csv columns to nested json

Question

Python 3.8.5 with Pandas 1.1.3

I have a csv file with columns: name, city, state, and zipcode. I need to convert to json with the city, state, and zipcode column values inside an object called residence.

For example:

CSV file

Name        City    State  Zipcode
John Doe    Akron   OH     44140

I need the JSON output to be structured like:

{
    "name": "John Doe",
    "residence" : 
        {
        "city": "Akron",
        "state": "OH",
        "zipcode": 44140
        }
}

I currently use Pandas to convert csv to json using the following code:

import pandas as pd

csv_file = pd.DataFrame(pd.read_csv("data.csv", sep = ",", header = 0, index_col = False))
csv_file.to_json("data.json", orient = "records", lines = True, date_format = "iso", double_precision = 10, force_ascii = True, date_unit = "ms", default_handler = None)

As-is, that just converts each column to a json key:value.

How can I add to this code to achieve my desired output?

Answer 1

IIUC try creating the nested object row-wise first, then creating the JSON:

import pandas as pd

csv_file = pd.read_csv("data.csv", sep=",",
                       header=0, index_col=False)

# Create Nested dict (Object)
csv_file['Residence'] = csv_file[['City', 'State', 'Zipcode']].apply(
    lambda s: s.to_dict(), axis=1
)
# Write out Name and Residence Only
csv_file[['Name', 'Residence']].to_json("data.json", orient="records",
                                        lines=True, date_format="iso",
                                        double_precision=10, force_ascii=True,
                                        date_unit="ms", default_handler=None)

data.csv

Name,City,State,Zipcode
John Doe,Akron,OH,44140
Jane Smith,Los Angeles,California,90005

data.json

{"Name":"John Doe","Residence":{"City":"Akron","State":"OH","Zipcode":44140}}
{"Name":"Jane Smith","Residence":{"City":"Los Angeles","State":"California","Zipcode":90005}}