Manipulating a list of dictionaries based on values in another list

Question

So basically I have three lists like so:

list1 = [{
    'IP' : "1.1.1.1", 
    'ID' : 1,
    },
    {
    'IP' : "2.2.2.2", 
    'ID' : 2
    }]

list2 = [{
    'vulnerability_id' : 4567, 
    'ID' : 1,
    },
    {
    'vulnerability_id' : 6578, 
    'ID' : 2
    }]

list3 = [{
    'vulnerability_id' : 4567, 
    'description' : 'blah',
    },
    {
    'vulnerability_id' : 6578, 
    'description' : 'blah blah'
    }]

What I need to do is get the description of an vulnerability based on an IP but this probably requires some sort of list comprehension and im not sure how to go about that. (And put it in a new dictionary)

I need to check the IP value I have, say 1.1.1.1, then compare it's ID with list2, then compare the associated 'vulnerability_id' in list3.

Any help would be greatly appreciated, if this doesn't make sense, please say and I will try and expand.

Answer 1

This is really heavy in terms of computing, but you only have lists as data sources, so, here is my "solution":

results = []
for ip_info in list1:
    result = {}
    result['ip'] = ip_info['IP']
    result['vulnerability'] = next((
            vuln_info['vulnerability_id']
            for vuln_info in list2
            if vuln_info['ID'] == ip_info['ID']
        ),None)

    result['description'] = next((
            desc_info['description']
            for desc_info in list3
            if desc_info['vulnerability_id'] == result['vulnerability']
        ),None)

    results.append(result)

Results:

[{'description': 'blah', 'ip': '1.1.1.1', 'vulnerability': 4567},
 {'description': 'blah blah', 'ip': '2.2.2.2', 'vulnerability': 6578}]

EDIT: improvement based on my answer and @Alex Hall answer:

def find(l, match, v, k):
    return next((x[k] for x in l if x[match] == v), None)

results = []
for ip_info in list1:
    ip = ip_info['IP']
    _id = ip_info['ID']
    vul = find(list2, 'ID', _id, 'vulnerability_id')
    desc = find(list3, 'vulnerability_id', vulnerability, 'description')    
    results.append(dict(ip=ip, vulnerability=vul, description=desc))

Answer 2

list1 = [
    {
        'IP': '1.1.1.1',
        'ID': 1,
    },
    {
        'IP': '2.2.2.2',
        'ID': 2
    }]

list2 = [
    {
        'vulnerability_id': 4567,
        'ID': 1,
    },
    {
        'vulnerability_id': 6578,
        'ID': 2
    }]

list3 = [
    {
        'vulnerability_id': 4567,
        'description': 'blah',
    },
    {
        'vulnerability_id': 6578,
        'description': 'blah blah'
    }]


def find(data, key, value):
    for row in data:
        if row[key] == value:
            return row
    raise ValueError('Row with %s = %s not found' % (key, value))


ID = find(list1, 'IP', '1.1.1.1')['ID']
vulnerability_id = find(list2, 'ID', ID)['vulnerability_id']
print(find(list3, 'vulnerability_id', vulnerability_id)['description'])

Answer 3

For things like this, I really like using the pandas package. I was able to do something like this:

import pandas as pd

list1 = [{
'IP' : "1.1.1.1", 
'ID' : 1,
},
{
'IP' : "2.2.2.2", 
'ID' : 2
}]

list2 = [{
'vulnerability_id' : 4567, 
'ID' : 1,
},
{
'vulnerability_id' : 6578, 
'ID' : 2
}]

list3 = [{
'vulnerability_id' : 4567, 
'description' : 'blah',
},
{
'vulnerability_id' : 6578, 
'description' : 'blah blah'
}]

df_1 = pd.DataFrame(list1)
df_2 = pd.DataFrame(list2)
df_3 = pd.DataFrame(list3)

output = df_1.merge(df_2).merge(df_3)

print output

ID       IP  vulnerability_id description
 1  1.1.1.1              4567        blah
 2  2.2.2.2              6578   blah blah

This will also give you that nicely formatted table at the end. With no extra work on your part! The merge operation is taking advantage of how well your dictionary keys are named. Since list1 and list2 both have id in common, it essentially performs an innerjoin on the id column. Then uses the resulting frame's vulnerability_id column to join with list3. This gets more complicated when you have more than one common key (in the sense that you will have to play around with some of the kwargs for the merge function to handle the excess overlap), but for this situation it works great :). The downside is that it does require installation of an external package, whereas the two solutions mentioned above do not.