How does python iterator for list actually works?

Question

Let's say we have the following list and we are creating an iterator for it:

lst = [1,2,3]
itr = iter(lst)

Next lets say we are changing our list with completely different values:

lst = ['a', 'b', 'c']

And if I we run the following loop:

for x in itr:
   print x

We will get '1,2,3' . But why? As far as I understand, iterator doesn't copy all values from iterating object. At least iterator for list from three elements has the same size as a list of 100000 elements. sys.getsizeof(i) returns 64 . How can iterator be so small by size and keep 'old' values of list?

Answer 1

The iterator itself contains a reference to the list. Since lst is rebound instead of mutated, this reference does not change.

>>> lst = [1, 2, 3]
>>> itr = iter(lst)
>>> lst[:] = ['a', 'b', 'c']
>>> for x in itr:
...   print x
... 
a
b
c

Answer 2

The iterator references a list object not a name. So reassigning the name lst to another object does not affect the iterator in anyway; names are bound to objects, and refer to objects, but the names are not the object themselves.

You can get a snoop of the object the iterator is referencing with gc.get_referents :

>>> import gc
>>> lst = [1,2,3]
>>> itr = iter(lst) # return an iterator for the list
>>> lst = ['a', 'b', 'c'] # Bind name lst to another object
>>> gc.get_referents(itr)[0]
[1, 2, 3]

As you'll notice, the iterator is still referring to the first list object.

---

The following reference will help you learn more about names and binding in Python:

Execution model - Naming and binding

Answer 3

Welcome to Python's object reference system. The variable names do not really have a deep relationship with the actual object stored in memory.

Suppose you have a friend lst , and you hire a mugger iter to mug him. Now you tell the mugger that your friend is the third Jack in the telephone directory ( globals ).

lst = [1, 2, 3]
itr = iter(lst)     # iter object now points to the list pointed to by lst
                    # it doesn't care about its name (doesn't even knows its name actually)

# Now the mugger has found the friend, and knows his address (the actual object in memory).
# The mugger catches him, and takes his jacket.
print itr.next()    # outputs 1

# Now the telephone directory was updated (yes it's updated very frequently).
lst1 = lst             # your friend is now the fourth Jack
lst = ['a', 'b', 'c']  # someone else is the third Jack now
                       # but mugger doesn't know, he won't see the directory again

print itr.next()       # (output 2), mugger takes t-shirt, and leaves him for now

# Meanwhile your friend buys new clothes.
lst1.append(4)      # here the actual object pointed to by iter is updated
lst1.append(5)

# You call the mugger and say, don't leave him until he's got nothing.
# The mugger goes full retard.
for i in iter:
    print i         # outputs 3, 4 and 5

NTL;DR : Python variable names are just tags, that refer to some object in space. When you call iter on the list named lst , the iterator object kind of gets the pointer to the actual object, and doesn't even now its name was lst .

If you can modify the original object, by calling append , extend , pop , remove , etc, the iterators behaviour will be affected. But when you assign a new value to lst , a new object is created (if it didn't previously exist), and lst simply starts pointing to that new object.

The garbage collector will delete the original object if no other object is pointing to it ( itr is pointing to it in this case, so the original object won't be deleted yet).

http://foobarnbaz.com/2012/07/08/understanding-python-variables/

Extra:

# The friend goes and buys more clothes.
lst1.extend([6, 7, 8])

# You call the mugger and ask him to take a shot at the friend again.
itr.next()    # But the mugger says, chill man he's got nothing now
              # raises StopIteration

This doesn't have anything to do with object reference, the iterator just stores internally that it has iterated the complete list.