How to search for key value pairs in a complex tree-like nested dict based on string values which contains a Jinja-like separators in Python

Question

I have a dictionary which contains nested dictionaries and nested lists of dictionaries. It goes few levels deeper then that. Some random keys will have values which will contain double hashes as a delimiters (same like Jinja's double curly braces). Those will be the entire string value of some random key. I need to figure out what is the key name for where I've found a value which contains the double hashes and what is the value in between.

I tried to think about a solution of translating the json into a string and simply look for the double hashes, like in 'sed' plus some string parsing, but I might loose the tractability for the key name, and this might get ugly. Trying to recourse over the master dict object did not help either because the data structure is complex, sometimes inside a dict there's a list and so on.

Example Dict:

{
    "key": 7,
    "networksCollection": [
        {
            "key": 47,
            "subnet": {
                "key": 73,
                "prefixDelegationEnabled": "##prefix##"
            },
            "providerSegmentationID": null,
            "providerPhysicalNetwork": "##physicalNetwork##"
        }

         ]
}

A flat dictionary as an output could be nice. something like:

{ "prefixDelegationEnabled": "prefix", "providerPhysicalNetwork": "physicalNetwork"}

Using this flat dict, and based on those keys it will contain, I will build an html form where the user will be prompted to populate values for those keys, where I will need to replace the double hashes place-holders with those values - meaning I will need a way to trace back to populate those place-holders

Answer 1

You can use a generator for that purpose which traverses the nested structure and yields key-value pairs for which the criterion is met:

def find_key_value_pairs(obj):
    if isinstance(obj, list):
        for item in obj:
            yield from find_key_value_pairs(item)
    elif isinstance(obj, dict):
        for key, value in obj.items():
            if isinstance(value, str) and value.startswith('##') and value.endswith('##'):
                yield key, value.strip('#')
            elif isinstance(value, (dict, list)):
                yield from find_key_value_pairs(value)

Then you can obtain the result via:

result = dict(find_key_value_pairs(test))

where test is your example dict.