根据列表中的第一个元素制作字典

Question

This is a question about performance of using set() on list comprehension inside dictionary comprehension Vs dictionary comprehension and repeated assignment to a new dictionary

So I happen to have a dataset which is a list of lists and i need to get a unique list of elements that are indexed at '0' in each of those lists inside the big list, so as to be able to make a new dictionary from them.. something like dict.fromkeys() .. here I need to supply list of unique keys..

I'm using

[1] { x : [] for x in set([i[0] for i in data])}

and also using

[2] { i[0] : [] for i in data}

sample data for reference here could be like: [[1,3,4], [3,5,2], [1,5,2]]

the result from running [1] and [2] above would then be: { 1:[], 3: [] }

I tried %timeit on both statements and both give nearly same results which makes it difficult to identify which one is best, performance-wise, for big list of lists

How do I identify a potential bottleneck here?

EDIT:

If this helps in explaining the results..

In [172]: data_new = data * 10000

In [173]: %timeit { i[0] : [] for i in data_new}
10 loops, best of 3: 160 ms per loop

In [174]: %timeit { x : [] for x in set([i[0] for i in data_new])}
10 loops, best of 3: 131 ms per loop

In [175]: data_new = data * 100000

In [176]: %timeit { x : [] for x in set([i[0] for i in data_new])}
1 loops, best of 3: 1.37 s per loop

In [177]: %timeit { i[0] : [] for i in data_new}
1 loops, best of 3: 1.58 s per loop

In [178]: data_new = data * 1000000

In [179]: %timeit { i[0] : [] for i in data_new}
1 loops, best of 3: 15.8 s per loop

In [180]: %timeit { x : [] for x in set([i[0] for i in data_new])}
1 loops, best of 3: 13.6 s per loop

Answer 1

Build a larger dataset, then timeit:

Code:

import random
data = [ [random.randint(1, 9) for _ in range(3)] for _ in range(1000000)]

Timings:

%timeit { x : [] for x in set([i[0] for i in data])}
# 10 loops, best of 3: 94.6 ms per loop
%timeit { i[0] : [] for i in data}
# 10 loops, best of 3: 106 ms per loop
%timeit { x: [] for x in set(i[0] for i in data)}
# 10 loops, best of 3: 114 ms per loop
%timeit { x: [] for x in {i[0] for i in data}}
# 10 loops, best of 3: 77.7 ms per loop

Rationale :

Limiting the available key space first means the dictionary only has to assign (given the randint above) 9 unique keys to 9 new lists. When using a dict comp, the dictionary has to repeatedly create and re-assign the value of its key to a newly created list... The difference is overhead in the deallocation of discarded empty lists (being garbage collected) and the time spent to create a new empty list.

Given a uniform distribution from randint , then there's 111,111 allocations and de-allocations of empty lists for 9 unique values over a set of 1,000,000 elements -- that's a lot more than just 9.

Answer 2

It depends on how many duplicates you expect. In the shorter code, the empty list is created for each item in the input list, and this is surprisingly expensive. Use a static value, and the shorter becomes faster.

In the following, L = [[1,3,4], [3,5,2], [1,5,2]] * 100000

In [1]: %timeit { x : [] for x in {i[0] for i in L]}}
10 loops, best of 3: 58.9 ms per loop

In [2]: %timeit { i[0] : [] for i in L}
10 loops, best of 3: 68.1 ms per loop

Now test with the constant None value here:

In [3]: %timeit { x : None for x in set([i[0] for i in L])}
10 loops, best of 3: 59 ms per loop

In [4]: %timeit { i[0] : None for i in L}
10 loops, best of 3: 54.3 ms per loop

Thus the needless list creation makes the shorter one perform slowly, whereas it is absolutely faster with constant values.

---

I didn't have ipython for Python 2 and I am a bit lazy timing this, but you would want to notice that Python 2.7 supports set comprehensions , which at least on Python 3.4 are much faster than creating sets from lists:

In [7]: %timeit { x : [] for x in {i[0] for i in L}}
10 loops, best of 3: 48.9 ms per loop