Iterating through a dictionary using values from a list

Question

I'm working with some data that has ever-changing keys for each record. I've collected a master list of keys from every record and stored them in a list. I'd like to then iterate through the list to pull the appropriate values using a try catch but I'm getting an error due to the item in the list item being a str and the dictionary being a dict .

An example, notice the possibility of nested keys. Otherwise my approach would be much simpler:

records = [{
  "id": 2017215,
  "name": "foo bar"
  "campaign": {
    "id": 161,
    "name": "Historical Data Campaign"
  }
},
{
  "id": 2017215,
  "name": "foo bar",
  "last_updated": "2018-01-01",
  "campaign": {
    "id": 161,
    "name": "Historical Data Campaign"
  }
}]

keys = [ ['id'], ['name'], ['campaign'], ['campaign']['id'], ['campaign']['name'], ['last_updated'] ]

for record in records:
  for key in keys:
    print (record + key) # assumption was this would generate "record['id']" but errors due to mismatched types.

Am I making this harder than it needs to be? The biggest issue is the keys can change from record to record, and in some cases there are nested keys in the data.

Answer 1

First, you can make the above snippet work if you did something along the lines of

records = # ...
keys = [ ('id',), ('name',), ('campaign',), ('campaign', 'id'), ('campaign', 'name'), ('last_updated',) ]

for record in records:
    for key_tuple in keys:
        obj = record
        for key in key_tuple:
            obj = record[key]
        print(obj)

One issue from your snippet is the occurrence of elements like ['campaign']['id'] . Python assumes the ['campaign'] part is a literal list definition, and then the ['id'] is indexing that list (but can't because it's not an integer).

More generally, I would suggest looking into the .items() method for dictionaries. You could turn the above snippet into

for record in records:
    for key, value in record.items():

        # to handle nested dictionaries
        if isinstance(value, dict):
            for subkey, subval in value.items():
                print(subkey, subval)
        else:
            print(key, val)

That will handle any keys, but if you have deeper than two levels of nested dictionaries, you may want to re-implement it as a recursive function.