Writing values from dictionaries to CSV in Python

Question

I'm reading the key: value pair from the list of dictionaries in a JSON file. The problem is I don't know how to write this in a CSV file. The code I have tried so far is:

import csv
import pandas as pd
import json
from pandas.io.json import json_normalize
import pyarrow as pa
import pyarrow.parquet as pq


arxml_json = open("C:\\Users\\abc\\xyz\\parser.json")
dataList = json.load(arxml_json)

# print(Results)
csv_file = "output.csv"
csv_columns = ['message_id', 'is_extended_frame','name', 'length', 'bit_length', 'factor', 'is_big_endian','is_signed,name', 'offset', 'start_bit', 'minimum', 'maximum', 'unit', 'recv_nodes', 'val_desc']

with open(csv_file, 'w') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
    writer.writeheader()

for key in dataList['messages']:
    id = key['id']
    is_extended_frame = key['is_extended_frame']
    name = key['name']
    length = key['length']
    for value in key['signals']:
        bit_length = value['bit_length']
        factor = value['factor']
        is_big_endian = value['is_big_endian']
        #is_float = value['is_float']
        is_signed = value['is_signed']
        name = value['name']
        offset= value['offset']
        start_bit = value['start_bit']
        minimum = value['minimum']
        maximum = value['maximum']
        unit= value['unit']
        recv_nodes = value['recv_nodes']
        val_desc = value['val_desc']

The Json file looks like this:

"messages": [
        {
            "id": 666,
            "is_extended_frame": false,
            "name": "uptime",
            "length": 8,
            "signals": 
                [{
                "bit_length": 64,
                "factor": 1,
                "is_big_endian": true,
                "is_float": true,
                "is_signed": true,
                "name": "uptime",
                "offset": 98,
                "start_bit": 40,
                "minimum": 30,
                "maximum": 70,
                "unit": "m/s",
                "recv_nodes": "none",
                "val_desc": "-none-"
                },{
                "bit_length": 64,
                "factor": 1,
                "is_big_endian": true,
                "is_float": true,
                "is_signed": true,
                "name": "uptime",
                "offset": 98,
                "start_bit": 40,
                "minimum": 30,
                "maximum": 70,
                "unit": "m/s",
                "recv_nodes": "none",
                "val_desc": "-none-"
                }]

        },

I want to create a CSV file that will look like below:

Any leads on how can I achieve this? Thanks in advance.

Answer 1

You can iterate over all of the dict s in the messages list inside of the dict returned by your json.load call and collapse the signals list so that each key in the sub- dict becomes a key in the overall dict so you don't have to explicitly pull out each property:

with open('test.json') as arxml_json:
    dataList = json.load(arxml_json)

# print(Results)
csv_file = "output.csv"
csv_columns = ['message_id', 'is_extended_frame', 'message_name', 'length', 'bit_length', 'factor', 'is_big_endian','is_signed', 'signal_name', 'offset', 'start_bit', 'minimum', 'maximum', 'unit', 'recv_nodes', 'val_desc']

with open(csv_file, 'w') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
    writer.writeheader()

    for message in dataList['messages']:
        for signals in message['signals']:
            temp = {k: v for k, v in message.items() if k != 'signals'}
            temp['message_name'] = temp['name']
            temp['message_id'] = temp['id']
            del temp['name']
            del temp['id']
            for key, value in signals.items():
                    if key == 'name':
                        temp['signal_name'] = value
                    else:
                        temp[key] = value
            del temp['is_float'] # don't care about this field
            print(temp)
            writer.writerow(temp)

What this is doing is first, iterating through all of the messages in your JSON, and then iterating through all of the possible dict s in message['signals'] . During each iteration over signals , temp is initially created as a copy of the message dictionary, excluding the signals list , because we want to copy those items out of the list . Then, it renames some of the properties according to the requirements in your screenshot: name in the top-level becomes message_name and id becomes message_id . it uses the del keyword to remove the key-value pair from the dict once it has renamed it. Then, for all of the key-value pairs in the current signal dict , it assigns them to temp . Finally, it deletes the field which you don't show in your screenshot and then writes the temp dict to the file

Answer 2

the nature of comma separated value (csv) file is a long string that separate certain value with comma.

1. create a string that separated with value

data_to_write = []
data_to_write.append("id_no")
data_to_write.append("is_extended_frame_bool")
# and so on

# join all the content in the data_to_write_list to one long string
string_to_append = ",".join(map(str,data_to_write))

2. append the string to csv file

output_file_name = "mydata.csv"
with open(output_file_name,"a") as file_object:
    file_object.write(string_to_append +"\n")

# if "\n" not be use, it will become a long line