Recursively finding paths in a nested Dict

Question

I have a nested dictionary:

d = {
 "@timestamp": "2019-01-08T19:33:50.066Z",
 "metricset": {
    "rtt": 2592,
    "name": "filesystem",
    "module": "system"
 },
 "system": {
    "filesystem": {
        "free_files": 9660022,
        "type": "rootfs",
        "device_name": "rootfs",
        "available": 13555355648,
        "files": 9766912,
        "mount_point": "/",
        "total": 19992150016,
        "used": {
            "pct": 0.322,
            "bytes": 6436794368
        },
        "free": 13555355648
    }
 },
 "host": {
    "name": "AA"
 },
 "beat": {
    "name": "AA",
    "hostname": "AA",
    "version": "6.3.2"
  }
}

What I would like to do is write this dictionary to a CSV file. I'd like the headers of the csv to be something like this:

system.filesystem.type

where the path is made up by each level separated by a period. I am able to go through the dictionary and get most of the headers I need; however my problem is with duplicate values.

PROBLEM: I recursively go through the dict and grab all the values and put them in a list. Then, I am searching for those values in the dictionary again, but this time saving the path to construct the header. However, with duplicate values (ie the value "rootfs"), I am getting only the first key-value ("type": "rootfs") returned.

Here is my traversal to grab all the values from the dict, which does exactly what I want:

def traverse(valuelist, dictionary):

    for k,v in dictionary.items():

       if isinstance(v, dict):
         traverse(valuelist,v)
       else:
         valuelist.append(v)

    return valuelist

Now here is the code that grabs the path for each value from the code above:

def getpath(nested_dict, value, prepath=()):
    for k,v in nested_dict.items():
        path = prepath + (k,)
        if v == value: # found value
           return path
        elif hasattr(v, 'items'): # v is a dict
            p = getpath(v, value, path) # recursive call
            if p is not None:
                return p

This part is not my own code. I found it here on SO and would like to modify it to grab every unique path for duplicate values (ie For value "rootfs" 1st path: "system.filesystem.type" 2nd path: "system.filesystem.device_name").

Thank you very much, and any help is appreciated!

Answer 1

An easy way to do this is to turn getpath into a generator:

def getpath(nested_dict, value, prepath=()):
    for k,v in nested_dict.items():
        path = prepath + (k,)
        if v == value: # found value
            yield path # yield the value
        elif hasattr(v, 'items'):
            yield from getpath(v, value, path) # yield all paths from recursive call

This way it yields every single valid path recursively. You can use it like so:

for path in getpath(nested_dict, value):
    # do stuff with path