Reading/writing a list of nested dictionaries to/from a CSV file (Python)

Question

I have a data structure that looks like this:

data  =[
{'key_1': { 'calc1': 42, 'calc2': 3.142 } },
{'key_2': { 'calc1': 123.4, 'calc2': 1.414 } },
{'key_3': { 'calc1': 2.718, 'calc2': 0.577 } }
]

I want to be able to save/and load the data into a CSV file with the following format

key,    calc1,   calc2   <- header
key_1,  42,      3.142   <- data rows
key_2,  123.4,   1.414
key_3,  2.718,   0.577

What's the 'Pythonic' way to read/save this data structure to/from a CSV file like the one above?

Answer 1

Just to show a version that does use the csv module:

from csv import DictWriter

data  =[
{'key_1': { 'calc1': 42, 'calc2': 3.142 } },
{'key_2': { 'calc1': 123.4, 'calc2': 1.414 } },
{'key_3': { 'calc1': 2.718, 'calc2': 0.577 } }
]

with open('test.csv', 'wb') as f:
    writer = DictWriter(f, ['key', 'calc1', 'calc2'])
    writer.writerow(dict(zip(writer.fieldnames, writer.fieldnames))) # no automatic header :-(
    for i in data:
        key, values = i.items()[0] # each dict in data contains only one entry
        writer.writerow(dict(key=key, **values)) # first make a new dict merging the key and the values

Answer 2

I don't think that it would be possible to use csv module, due to all the idiosyncrasies in your requirements and the structure, but you could do it quite easily writing it manually:

>>> with open('test.txt', 'w') as f:
    f.write(','.join(['key', 'calc1', 'calc2']) + '\n')
    f.writelines('{},{},{}'.format(k, *v.values()) + '\n' for l in data for k,v in l.items())

Answer 3

I feel the by far easiest approach is to transform the dictionary first into a pandas dataframe and from there very convenient into a csv file.

import pandas as pd    
df = pd.DataFrame.from_dict(data, orient='index') # it is either ‘columns’ or ‘index’ (simple transpose)
    df.to_csv('filepath/name.csv')

To load the file back into the dict form

df.to_csv('filepath/name.csv')
data_dict = df.to_dict()

For more complicated cases have a look into the pandas documentation http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html and http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_dict.html