How to efficiently insert key-value from one list of dictionaries to another based on a common key-value pair in Python?

Question

I have this big list of dictionaries in python. An example of which is below:

big_list_dictionary = [{
    'name': 'test = 1',
    'id': 1,
    'value': 30
},{
    'name': 'apple = 1',
    'id': 2,
    'value': 70
},{
    'name': 'orange = 1',
    'id': 3,
    'value': 10
},{
    'name': 'balloon = 1',
    'id': 4,
    'value': 20
},{
    'name': 'airplane = 1',
    'id': 5,
    'value': 40
}]

and I have a list of two dictionaries with their total values

total1 = [{
    'name': 'test',
    'total': 130
},{
    'name': 'apple',
    'total': 270
},{
    'name': 'orange',
    'total': 310
},{
    'name': 'balloon',
    'total': 420
},{
    'name': 'airplane',
    'total': 540
}]

total2 = [{
    'name': 'test',
    'total': 230
},{
    'name': 'apple',
    'total': 570
},{
    'name': 'orange',
    'total': 3210
},{
    'name': 'balloon',
    'total': 620
},{
    'name': 'airplane',
    'total': 940
}]

If you notice, the name in total1 and total2 is just slightly different than big_list_dictionary where = 1 is omitted.

How can I add the total values from total1 and total2 to big_list_dictionary so that the final result looks like the following:

[{
    'name': 'test = 1',
    'id': 1,
    'value': 30,
    'total2': 230,
    'total1': 130
},{
    'name': 'apple = 1',
    'id': 2,
    'value': 70,
    'total2': 570,
    'total1': 270
},{
    'name': 'orange = 1',
    'id': 3,
    'value': 10,
    'total2': 3210,
    'total1': 310
},{
    'name': 'balloon = 1',
    'id': 4,
    'value': 20,
    'total2': 620,
    'total1': 420
},{
    'name': 'airplane = 1',
    'id': 5,
    'value': 40,
    'total2': 940,
    'total1': 540
}]

Currently, the way I'm doing it is very slow.

    for item in big_list_dictionary:
        for t1,t2 in zip(total1,total2):
            if t1['name'] in item['name']:
                item['total1] = t1['total']
                item['total2'] = t2['total']

How can I do this efficiently?

Answer 1

If the extra characters are always =1 then you can create an intermediate relation and then use the following code.

big_list_dictionary = [{'name': 'test = 1','id': 1,'value': 30},{'name': 'apple = 1','id': 2,'value': 70},{'name': 'orange = 1','id': 3,'value': 10},{'name': 'balloon = 1','id': 4,'value': 20},{'name': 'airplane = 1','id': 5,'value': 40}]

total1 = [{ 'name': 'test','total': 130},{'name': 'apple','total': 270},
{'name': 'orange','total': 310},{'name': 'balloon','total': 420},{'name': 'airplane','total': 540}]
total2 = [{'name': 'test','total': 230},{'name': 'apple','total': 570},{'name': 'orange','total': 3210},{'name': 'balloon','total': 620},{'name': 'airplane','total': 940}]


intermediate = {i['name'].split('=')[0].strip():i for i in big_list_dictionary}

for t1, t2 in zip(total1, total2):
    intermediate[t1['name']]['total1'] = t1['total']
    intermediate[t1['name']]['total2'] = t2['total']

print(big_list_dictionary)

Output

[{'name': 'test = 1', 'id': 1, 'value': 30, 'total1': 130, 'total2': 230},
 {'name': 'apple = 1', 'id': 2, 'value': 70, 'total1': 270, 'total2': 570},
 {'name': 'orange = 1', 'id': 3, 'value': 10, 'total1': 310, 'total2': 3210},
 {'name': 'balloon = 1', 'id': 4, 'value': 20, 'total1': 420, 'total2': 620},
 {'name': 'airplane = 1', 'id': 5, 'value': 40, 'total1': 540, 'total2': 940}]

Benchmark

%%timeit -n10 -r10 with big_list_dictionary , total1 and total2 of length 1000 each. Length has been increased to show the efficiency.

This Solution

513 µs ± 17.2 µs per loop (mean ± std. dev. of 10 runs, 10 loops each)

Your Solution

91.6 ms ± 1.19 ms per loop (mean ± std. dev. of 10 runs, 10 loops each)

This solution is faster. ( 513 µs < 91.6 ms ) The difference between both the solutions will only increase as the length increases.

Edit based on comments:

I believe there are some elements in test1 and test2 that are not in big_list_dictionary , as there is an assumption that test1 and test2 have the same elements in the same order, you can just iterate through one of the list and append if not found to big_list_dictionary and also add that to intermediate . This appends all the new dicts at the end of big_list_dictionary , adding at the end is faster than inserting at a random position. But if you do care about order, then I believe your solution is as good as it gets.

Disclaimer: I have not tested this part of the code as I dont have the input or output to check the desired behavior.

big_list_dictionary = [{'name': 'test = 1','id': 1,'value': 30},{'name': 'apple = 1','id': 2,'value': 70},{'name': 'orange = 1','id': 3,'value': 10},{'name': 'balloon = 1','id': 4,'value': 20},{'name': 'airplane = 1','id': 5,'value': 40}]
total1 = [{ 'name': 'test','total': 130},{'name': 'apple','total': 270},{'name': 'orange','total': 310},{'name': 'balloon','total': 420},{'name': 'airplane','total': 540}]
total2 = [{'name': 'test','total': 230},{'name': 'apple','total': 570},{'name': 'orange','total': 3210},{'name': 'balloon','total': 620},{'name': 'airplane','total': 940}]


intermediate = {i['name'].split('=')[0].strip():i for i in big_list_dictionary}


for t1 in total1:
    if t1['name'] not in intermediate:         
        temp_dict = {'name': t1['name'],'id': 0,'value': 0}
        big_list_dictionary.append(temp_dict)
        intermediate[t1['name']] = temp_dict


# insert rest of the answer from code above