How to create a hierarchical dictionary from a csv?

Question

I am trying to build a hierarchical dict (please see below the desired output I am looking for) from my csv file.

The following is my code so far, I was searching through itertools possibly I think that's the best tool I need for this task. I cannot use pandas . I think I need to maybe put the values of the key into a new dictionary and then try to map the policy interfaces and build a new dict ?

import csv
import pprint
from itertools import groupby

new_dict=[]
with open("test_.csv", "rb") as file_data:
    reader = csv.DictReader(file_data)

    for keys, grouping in groupby(reader, lambda x: x['groupA_policy']):
        new_dict.append(list(grouping))

pprint.pprint(new_dict)

My csv file looks like this:

GroupA_Host,groupA_policy,groupA_policy_interface,GroupB_Host,GroupB_policy,GroupB_policy_interface
host1,policy10,eth0,host_R,policy90,eth9
host1,policy10,eth0.1,host_R,policy90,eth9.1
host2,policy20,eth2,host_Q,policy80,eth8
host2,policy20,eth2.1,host_Q,policy80,eth8.1

The desired output I want achieve is this:

[{'GroupA_Host': 'host1',
  'GroupB_Host': 'host_R',
  'GroupB_policy': 'policy90',
  'groupA_policy': 'policy10',
  'interfaces': [{'GroupB_policy_interface': 'eth9',
                  'group_a_policy_interfaces': 'eth0'},
                 {'GroupB_policy_interface': 'eth9.1',
                  'group_a_policy_interface': 'eth0.1'}]},
 {'GroupA_host': 'host2',
  'GroupB_Host': 'host_Q',
  'GroupB_policy': 'policy80',
  'groupA_policy': 'policy20',
  'interfaces': [{'GroupB_policy_interface': 'eth8',
                  'groupA_policy_interfaces': 'eth2'},
                 {'groupA_policy_interface': 'eth8.1',
                  'groupA_policy_interfaces': 'eth2.1'}]}]

Answer 1

I don't think itertools is necessary here. The important thing is to recognize that you're using ('GroupA_Host', 'GroupB_Host', 'groupA_policy', 'GroupB_policy') as the key for the grouping -- so you can use a dictionary to collect interfaces keyed on this key:

d = {}

for row in reader:
    key = row['GroupA_Host'], row['GroupB_Host'], row['groupA_policy'], row['GroupB_policy']
    interface = {'groupA_policy_interface': row['groupA_policy_interface'], 
                 'GroupB_policy_interface': row['GroupB_policy_interface']
    }

    if key in d:
        d[key].append(interface)
    else:
        d[key] = [interface]

as_list = []
for key, interfaces in d.iteritems():
    record = {}
    record['GroupA_Host'] = key[0]
    record['GroupB_Host'] = key[1]
    record['groupA_policy'] = key[2]
    record['GroupB_policy'] = key[3]
    record['interfaces'] = interfaces
    as_list.append(record)