How can I convert a CSV file to a JSON file with a nested json object using Python?

Question

I am stuck with a problem where I don't know how I can convert a "nested JSON object" inside a CSV file into a JSON object.

So I have a CSV file with the following value:

data.csv

1, 12385, {'message': 'test 1', 'EngineId': 3, 'PersonId': 1, 'GUID': '0ace2-02d8-4eb6-b2f0-63bb10829cd4s56'}, 6486D, TestSender1
2, 12347, {'message': 'test 2', 'EngineId': 3, 'PersonId': 2, 'GUID': 'c6d25672-cb17-45e8-87be-46a6cf14e76b'}, 8743F, TestSender2

I wrote a python script that converts this CSV file into a JSON file inside an array. This I did with the following python script

csvToJson.py

import json
import csv


with open("data.csv","r") as f:
    reader = csv.reader(f)
    data = []
    for row in reader:
        data.append({"id": row[0],
        "receiver": row[1],
        "payload": row[2],
        "operator": row[3],
        "sender": row[4]})

with open("data.json", "w") as f:
    json.dump(data, f, indent=4)

The problem I'm facing is that I'm not getting the right values inside "payload", which I would like to be a nested JSON object. The result I get is the following:

data.json

[
    {
        "id": "1",
        "receiver": " 12385",
        "payload": " {'message': 'test 1'",
        "operator": " 'EngineId': 3",
        "sender": " 'PersonId': 1"
    },
    {
        "id": "2",
        "receiver": " 12347",
        "payload": " {'message': 'test 2'",
        "operator": " 'EngineId': 3",
        "sender": " 'PersonId': 2"
    }
]

So my question is, how can I create a nested JSON object for the "payload" while I'm doing the conversion from CSV to JSON? I think the main problem is that it is seen as a string and not as an object.

Answer 1

Try the following. You can just do everything as previously, but merge back all elements that were in 3rd column and load it via ast.literal_eval .

import json
import csv
import ast    

with open("data.csv","r") as f:
    reader = csv.reader(f,skipinitialspace=True)
    data = [{"id": ident,
             "receiver": rcv,
             "payload": ast.literal_eval(','.join(payload)),
             "operator": op,
             "sender": snd}
            for ident,rcv,*payload,op,snd in reader]

with open("data.json", "w") as f:
    json.dump(data, f, indent=4)