Transform nested Python dictionary to get same-level key values on the same row in CSV output

Question

The following code represents a sample a log data I'm trying to transform and export to CSV. It can either have a nested dict for warning and error (ex: agent 1) or have no dict for warning or error (ex: agent 2).

dlog = \
[
    {
        'agentName': 'agent 1',
        'date': '2018-03-26',
        'fileName': 'log_2018-3-26.log',
        'warning':
            {
                'Street': 618,
                'Suite': 470,
                'TargetID': 558,
                'Error loading page frame': 27,
                'writeOverride': 51,
                'State': 53,
                'Zip': 52,
                'PhoneNumber': 5
            },
        'error':
            {
                'Locations error handling': 1,
                'The web browsers are not responding': 13
            }
    },
    {
        'agentName': 'agent 2',
        'date': '2018-03-26',
        'fileName': 'log_2018-3-26.log'
    }
]

with open('C:\\users\\dbrewer\\documents\\LastLogTest.csv', 'w', newline='') 
as csvOutput:
    fieldnames = ['agentName', 'date', 'fileName', 'warnings', 'warningCount', 'errors', 'errorCount']
    dict_writer = csv.DictWriter(csvOutput, fieldnames)
    dict_writer.writeheader()
    for d in dlog:
        dict_writer.writerow(d)

This is the output I'm getting:

As you can see, I'm not breaking down the nested dict for warning and error the way I want too. I've been able to produce a dataset that turns the nested dict values into columns.

This is partially correct

However,this will not work for the working data because there are dozens of different warning and error keys, some of considerable length. What I need is to export the related key, value pairs (agentName, date, fileName) for each key, value pair in the nested warning and error dicts. This would produce 9 rows and 7 columns for the dataset (below)

This is the output I need:

Obviously there is a solution for this, but all I've been able to find here is examples transforming the nested keys into columns (not rows). What's the best approach for tacking this type of transformation?

Answer 1

You can create a function to flatten the structure:

import itertools
import csv
def results(d):
  start = [[d[i] for i in ['agentName', 'date', 'fileName']]+list(c) for c in d.get('warning', {'':None}).items()]
  return [filter(None, a+list(b)) if b else a for a, b in itertools.izip_longest(start, d.get('error', {'':None}).items())]

with open('filename.csv', 'w') as f:
   write = csv.writer(f)
   write.writerow(['agentName', 'date', 'fileName', 'warnings', 'warningCount', 'errors', 'errorCount'])
   for data in map(results, dlog):
      write.writerows(data)

Output:

agentName,date,fileName,warnings,warningCount,errors,errorCount
agent 1,2018-03-26,log_2018-3-26.log,writeOverride,51,Locations error handling,1
agent 1,2018-03-26,log_2018-3-26.log,State,53,The web browsers are not responding,13
agent 1,2018-03-26,log_2018-3-26.log,Street,618
agent 1,2018-03-26,log_2018-3-26.log,PhoneNumber,5
agent 1,2018-03-26,log_2018-3-26.log,Zip,52
agent 1,2018-03-26,log_2018-3-26.log,Error loading page frame,27
agent 1,2018-03-26,log_2018-3-26.log,Suite,470
agent 1,2018-03-26,log_2018-3-26.log,TargetID,558
agent 2,2018-03-26,log_2018-3-26.log