Improving Python Snippet Performance

Question

This statement is running quite slowly, and I have run out of ideas to optimize it. Could someone help me out?

[dict(zip(small_list1, small_list2)) for small_list2 in really_huge_list_of_list]

The small_list s contain only about 6 elements.

A really_huge_list_of_list of size 209,510 took approximately 16.5 seconds to finish executing.

Thank you!

Edit:

really_huge_list_of_list is a generator. Apologies for any confusion. The size is obtained from the result list.

Answer 1

Possible minor improvement:

[dict(itertools.izip(small_list1, small_list2)) for small_list2 in really_huge_list_of_list]

Also, you may consider to use generator instead of list comprehensions.

Answer 2

To expand on what the comments are trying to say, you should use a generator instead of that list comprehension. Your code currently looks like this:

[dict(zip(small_list1, small_list2)) for small_list2 in really_huge_list_of_list]

and you should change it to this instead:

def my_generator(input_list_of_lists):
    small_list1 = ["wherever", "small_list1", "comes", "from"]
    for small_list2 in input_list_of_lists:
        yield dict(zip(small_list1, small_list2))

What you're doing right now is taking ALL the results of iterating over your really huge list, and building up a huge list of the results, before doing whatever you do with that list of results. Instead, you should turn that list comprehension into a generator so that you never have to build up a list of 200,000 results. It's building that result list that's taking up so much memory and time.

... Or better yet, just turn that list comprehension into a generator comprehension by changing its outer brackets into parentheses:

(dict(zip(small_list1, small_list2)) for small_list2 in really_huge_list_of_list)

That's really all you need to do. The syntax for list comprehensions and generator comprehensions is almost identical, on purpose: if you understand a list comprehension, you'll understand the corresponding generator comprehension. (In this case, I wrote out the generator in "long form" first so that you'd see what that comprehension expands to).

For more on generator comprehensions, see here , here and/or here .

Hope this helps you add another useful tool to your Python toolbox!