__iter__() implemented as a generator

Question

I have an object subclass which implements a dynamic dispatch __ iter __ using a caching generator (I also have a method for invalidating the iter cache) like so:

def __iter__(self):
    print("iter called")
    if self.__iter_cache is None:
        iter_seen = {}
        iter_cache = []
        for name in self.__slots:
            value = self.__slots[name]
            iter_seen[name] = True
            item = (name, value)
            iter_cache.append(item)
            yield item           
        for d in self.__dc_list:
            for name, value in iter(d):
                if name not in iter_seen:
                    iter_seen[name] = True
                    item = (name, value)
                    iter_cache.append(item)
                    yield item
        self.__iter_cache = iter_cache
    else:
        print("iter cache hit")
        for item in self.__iter_cache:
            yield item

It seems to be working... Are there any gotchas I may not be aware of? Am I doing something ridiculous?

Answer 1

container.__iter__() returns an iterator object. The iterator objects themselves are required to support the following two methods, which together form the iterator protocol:

iterator.__iter__()

Returns the iterator object itself.

iterator.next()

Return the next item from the container.

That's exactly what every generator has. So don't afraid any side-effects.

Answer 2

It might be better to separate the iteration of the object from the caching of the values it returns. That would simplify the iteration process and allow you to easily control how the caching is accomplished as well as whether it is enabled or not, for example.

Another possibly important consideration is the fact that your code would not predictively handle the situation where the object being iterated over gets changed between successive calls to the method. One simple way to deal with that would be to populate the cache's contents completely on the first call, and then just yield what it contains for each call -- and document the behavior.

Answer 3

It seems like a very fragile approach. It is enough to change any of __slots, __dc_list, __iter_cache during active iteration to put the object into an inconsistent state.

You need either to forbid changing the object during iteration or generate all cache items at once and return a copy of the list.

Answer 4

What you're doing is valid albeit weird. What is a __slots or a __dc_list ?? Generally it's better to describe the contents of your object in an attribute name, rather than its type (eg: self.users rather than self.u_list).

You can use my LazyProperty decorator to simplify this substantially.

Just decorate your method with @LazyProperty. It will be called the first time, and the decorator will then replace the attribute with the results. The only requirement is that the value is repeatable; it doesn't depend on mutable state. You also have that requirement in your current code, with your self.__iter_cache.

def __iter__(self)
    return self.__iter

@LazyProperty
def __iter(self)
    def my_generator():
        yield whatever
    return tuple(my_generator())