提高元组列表到dict的转换速度

Question

I have a list l consisting of tuples of length 5. The first four entries are strings, the last one is an integer. A dummy function to create such a list may look as follows:

import numpy as np
import uuid
def get_dummy_data(n=10000):
    l = []
    for i in range(n):
        name = np.random.choice(["Cat", "Dog", "Duck"], 1)[0]
        c_id = uuid.uuid4().hex
        t_id = uuid.uuid4().hex
        l.append((c_id, t_id, name, "canFly", 1))
        if np.random.random() < 0.8:
            l.append((c_id, t_id, name, "isHungry", 0))
    return l

Now this list l contains tuples which have identical first three elements but differ in the last two. This is exemplified here by appending the same tuple again with 80% chance but changing the last two elements.

The goal is to convert this list of length-5 tuples into a dictionary in which the key is the first entry of the tuple (c_id) and the value is structured like this (t_id, (name, {"isHungry":0})) or this: (t_id, (name, {"canFly":1, "isHungry":0})).

This can be achieved by the following loop:

res = {}
for y in l:
    if y[0] not in res:
        res[y[0]] = (y[1], (y[2], {y[3]: y[4]}))
    else:
        res[y[0]][1][1].update({y[3]: y[4]}) 

The question is now: can I make this faster? There might be more than two tuples in the list l with the same c_id (in contrast to the get_dummy_data function) and we cannot assume any order in l . I always have a bad feeling when doing an explicit for loop to fill a dict so I bet there is a good way to make this faster.

Answer 1

You can do basic micro-optimizations that also make your code more readable. A big one is not using some_dict.update({x:y}) instead of some_dict[x] = y . But here's some timing differences:

In [12]: %%timeit
    ...: res = {}
    ...: for y in data:
    ...:     if y[0] not in res:
    ...:         res[y[0]] = (y[1], (y[2], {y[3]: y[4]}))
    ...:     else:
    ...:         res[y[0]][1][1].update({y[3]: y[4]})
    ...:
100 loops, best of 3: 15.3 ms per loop

In [13]: %%timeit
    ...: res = {}
    ...: for a,b,c,d,e in data:
    ...:     if a not in res:
    ...:         res[a] = (b, (c, {d: e}))
    ...:     else:
    ...:         res[a][1][1][d] = e
    ...:
100 loops, best of 3: 11 ms per loop

Here it is with .update . Note, each y[...] is a method-call, which slows things down. But the biggest component of the time savings was avoiding the .update({...} . Note, that approach requires the creation of a whole dict object for no good reason:

In [18]: %%timeit
    ...: res = {}
    ...: for a,b,c,d,e in data:
    ...:     if a not in res:
    ...:         res[a] = (b, (c, {d: e}))
    ...:     else:
    ...:         res[a][1][1].update({d:e})
    ...:
100 loops, best of 3: 13.8 ms per loop

Answer 2

this kind of loop is generally slow:

res = {}
for y in l:
    if y[0] not in res:
        res[y[0]] = (y[1], (y[2], {y[3]: y[4]}))
    else:
        res[y[0]][1][1].update({y[3]: y[4]}) 

because you're testing if the key belongs to the dictionary twice and there's the if/else statement.

I would use the binding property of variables in lambda & unpacking (borrowed from juanpa answer):

import collections
res = collections.defaultdict(lambda : (b, (c, {d: e})))

for a,b,c,d,e in l:
    res[a][1][1][d] = e

if key isn't in dictionary defaultdict creates a key using the current value of a , b ..., (thanks to lambda evaluating the values when executing, not when declaring) saving the test and creating the proper key each time. Now the update part is a bit redundant but it still should be faster because there's no if/then test.

This solution is faster than juanpa (already good) answer on my machine (0.23 seconds vs 0.27 seconds). I would call that a good collaborating effort since my first version was slower.