Flatten a result from itertools.product

Question

I have some python code like

from itertools import product
myItems = product(*groups.values())

This gives me itertools.product object that, when I iterate through, looks like

(myCustomObject,myCustomObject,myCustomObject(myCustomObject,myCustomObject,myCustomObject))

How can I flatten this object so that it looks like

(myCustomObject,myCustomObject,myCustomObject,myCustomObject,myCustomObject,myCustomObject)

I want to iterate through the object and NOT put it in a list because the myItems object contains multiple-billions of records. What is the most effective way to do this?

Answer 1

The output from itertools.product is an object that will produce tuples:

>>> list(itertools.product('ABCD', 'XYZ'))
[('A', 'X'), ('A', 'Y'), ('A', 'Z'), ('B', 'X'), ('B', 'Y'), ('B', 'Z'), ('C', 'X'), ('C', 'Y'), ('C', 'Z'), ('D', 'X'), ('D', 'Y'), ('D', 'Z')]

Assuming you want to just flatten all the tuples that product produces, use chain :

>>> list(itertools.chain.from_iterable(itertools.product('ABCD', 'XYZ')))
['A', 'X', 'A', 'Y', 'A', 'Z', 'B', 'X', 'B', 'Y', 'B', 'Z', 'C', 'X', 'C', 'Y', 'C', 'Z', 'D', 'X', 'D', 'Y', 'D', 'Z']

If the objects fed to product are themselves nested tuples or lists, product will not recursively descend into them:

>>> list(itertools.product('ABCD', ['w', 'x',['y','z']]))
[('A', 'w'), ('A', 'x'), ('A', ['y', 'z']), ('B', 'w'), ('B', 'x'), ('B', ['y', 'z']), ('C', 'w'), ('C', 'x'), ('C', ['y', 'z']), ('D', 'w'), ('D', 'x'), ('D', ['y', 'z'])]

If you want to flatten a list of arbitrary depth, you need to do that recursively:

def flatten(container):
    for i in container:
        if isinstance(i, list) or isinstance(i, tuple):
            for j in flatten(i):
                yield j
        else:
            yield i

>>> list(flatten(itertools.product('ABCD', ['w', 'x',['y','z']])))     
['A', 'w', 'A', 'x', 'A', 'y', 'z', 'B', 'w', 'B', 'x', 'B', 'y', 'z', 'C', 'w', 'C', 'x', 'C', 'y', 'z', 'D', 'w', 'D', 'x', 'D', 'y', 'z']

Although I honestly cannot think of a use case for having a nested list of objects of varying depth to feed to product in the first place...