Python 3中的迭代器

Question

In Python 3 a lot of functions (now classes) that returned lists now return iterables, the most popular example being range . In this case range was made an iterable in Python 3 to improve performance, and memory efficiency (since you don't have to build a list anymore).

Other "new" iterables are map , enumerate , zip and the output of the dictionary operations dict.keys() , dict.values() and dict.items() . (There are probably more, but I don't know them).

Some of them ( enumerate and map ) have become probably more memory efficient by converting them into iterables. In Python 2.7 the others simply created lists of objects which were already in memory, so they would have been memory efficient.

Why then turn them into iterables which you have to convert to lists every time you want to sort them, etc.?

Answer 1

Several reasons:

1. The dictionary operations now return dictionary view objects ; these act as sets as well, giving you a far richer object to work with in your code. In Python 2 you'd have to use the dict.view*() methods to do the same.

2. The dictionary operations in Python 2 produced a new list object; that list object takes up memory too, even if the indices reference existing objects. There is another side effect here; the list indices increment the reference counts on all those dictionary contents, which impacts performance as well (and potentially flush the CPU cache).

3. zip() and map() could always work on any iterables, including generators, but would pull everything into a big list when applied. By turning them into generators in Python 3 they no longer auto-consume such iterables.

Note that enumerate() in Python 2 never returned a list, it has always returned an iterator.

You can always get the old Python 2 behaviour simply by applying list() on such objects. If you needed sorted items, you'd call sorted() on the iterable. But you now have the choice rather than have the list object forced on you.

And for most use cases in Python, you never really needed to have a whole list to begin with. You usually iterate over such results. Sorting them is not the most common use case, indexing them isn't either. So for most use cases, the change is a win, giving programmers the tools to produce more efficient code with just the standard functions and types.

Answer 2

If you want to sort them, the iterable needs to be turned into a list (which sorted will handle for you)... but how often are you going to sort an enumerate object, compared to how often you're going to just iterate over it? What about sorting the items of a dict, compared to just iterating over them?

If your API produces a lazy iterator or other lazy iterable, you can turn that into a list with roughly the same amount of effort it would have taken to skip the iterator and produce a list directly. On the other hand, if your API produces a list, there's no way to avoid holding all the items in memory at once. The iterator is more flexible.