Iterating through nested dict in a particular order with python3

Question

I have a nested dict which has lists as well as dict in it as given below.

m = {'abc': 
      {'bcd': [
               {'cde':'100','def':'200','efg':'300'},
               {'cde':'3000','def':'500','efg':'4000'}
              ], 
       'ghi': 
         {'mnc': [
                  {'xyz':'8827382','mnx':'e838','wyew':'2232'}
                 ]
         }
       }
     }

My requirement is to match mnx key and if the value is 'e838' then get the value of the other keys in that particular dict. So from the above example I may require the value of xyz key.

For this, I have created a recursive looping function as given below which is working. However my question is whether there is a better / easy way to do it. Also what can be done in the same code if I need to get all the values with key mnx. Thanks.

Note: I am converting an XML into dict with the help of jxmlease lib.

def iterate_dict(dict1,key1,val1):
    for key, value in dict1.items():
        if key == key1 and value == val1:
            return dict1
        if isinstance(value,list):
            for item1 in value:
                if isinstance(item1,dict):
                    for k,v in item1.items():
                        if k == key1 and v == val1:
                            return item1
        if isinstance(value,dict):
            for key,var in value.items():
                if key == key1 and var == val1:
                    return value
                else:
                    return iterate_dict(value,key1,val1)

Answer 1

You can sort of "flatten" the dict into a list of dicts and then query as necessary:

def flatten_dict(d):
    flattened = []
    current = {}
    for k, v in d.items():
        if isinstance(v, dict):
            flattened.extend(flatten_dict(v))
        elif isinstance(v, list):
            flattened.extend(sum((flatten_dict(v_d) for v_d in v), []))
        else:
            current[k] = v
    if len(current) > 0:
        flattened = [current] + flattened
    return flattened

def values_in_flattened(flattened, key):
    return list(filter(None, (d.get(key, None) for d in flattened))) or None

m = {'abc': {'bcd':[{'cde':'100','def':'200','efg':'300'},{'cde':'3000','def':'500','efg':'4000'}], 'ghi':{'mnc':[{'xyz':'8827382','mnx':'e838','wyew':'2232'}]}}}
mf = flatten_dict(m)
efg_vals = values_in_flattened(mf, 'efg')
print(mf)
print(efg_vals)

>>>
[{'xyz': '8827382', 'mnx': 'e838', 'wyew': '2232'}, {'def': '200', 'efg': '300', 'cde': '100'}, {'def': '500', 'efg': '4000', 'cde': '3000'}]
['300', '4000']

Answer 2

m['abc']['bcd'] + m['abc']['ghi']['mnc']

out:

[{'cde': '100', 'def': '200', 'efg': '300'},
 {'cde': '3000', 'def': '500', 'efg': '4000'},
 {'mnx': 'e838', 'wyew': '2232', 'xyz': '8827382'}]

you should build a list of dict to iterate, rather than use raw data.

Answer 3

This code does the searching using recursive generators, so it will yield all the solutions as it finds them.

When iterate_dict finds a dict with the desired (key, value) pair it calls filter_dict , which creates a new dict to contain the output. This new dict contains the items of the dict passed to filter_dict except that it filters out the desired (key, value) pair, it also filters out any lists or dicts that that dict may contain. However, iterate_dict will recursively process those lists or dicts looking for further matches. If you don't want iterate_dict to look for further matches, it's easy to modify the code so that it doesn't do that; please see below.

If you want to search for dicts that contain the desired key and don't care about the value associated with that key you can pass None as the val argument, or just omit that arg.

I've modified your data slightly so we can test the recursive search for further matches in a dict that contains a match.

def filter_dict(d, key):
    return {k: v for k, v in d.items() 
        if k != key and not isinstance(v, (dict, list))}

def iterate_dict(d, key, val=None):
    if key in d and (val is None or d[key] == val):
        yield filter_dict(d, key)
    yield from iterate_list(d.values(), key, val)

def iterate_list(seq, key, val):
    for v in seq:
        if isinstance(v, list):
            yield from iterate_list(v, key, val)
        elif isinstance(v, dict):
            yield from iterate_dict(v, key, val)

# test

data = {
    'abc': {
        'bcd': [
            {'cde':'100', 'def':'200', 'efg':'300'},
            {'cde':'3000', 'def':'500', 'efg':'4000'},
            {'abc': '1', 'mnx': '2', 'ijk': '3', 
                'zzz': {'xyzz':'44', 'mnx':'e838', 'yew':'55'}
            },
        ], 
        'ghi': {
            'mnc': [
                {'xyz':'8827382', 'mnx':'e838', 'wyew':'2232'}
            ]
        }
    }
}

for d in iterate_dict(data, 'mnx', 'e838'):
    print(d)

output

{'yew': '55', 'xyzz': '44'}
{'xyz': '8827382', 'wyew': '2232'}

Here's a search that looks for all dicts containing the 'mnx' key:

for d in iterate_dict(data, 'mnx'):
    print(d)

output

{'ijk': '3', 'abc': '1'}
{'xyzz': '44', 'yew': '55'}
{'wyew': '2232', 'xyz': '8827382'}

If you don't want each dict to be recursively searched for further matches once a match has been found in it, just change iterate_dict to:

def iterate_dict(d, key, val=None):
    if key in d and (val is None or d[key] == val):
        yield filter_dict(d, key)
    else:
        yield from iterate_list(d.values(), key, val)