Power Sets too slow python

Question

I am using the following function to find the subsets of a list L. However, when converting the output of the function powerset into a list it takes way too long. Any suggestion?

For clarification, this powerset function does not output the empty subset and the subset L itself (it is intentional).

My list L:

L = [0, 3, 5, 6, 8, 9, 11, 13, 16, 18, 19, 20, 23, 25, 28, 29, 30, 32, 33, 35, 36, 38, 42, 43, 44, 45, 49, 50, 51, 53, 54, 56, 57, 62, 63, 64, 65, 66, 67, 71, 76, 78, 79, 81, 82, 84, 86, 87, 90, 92, 96, 97, 98, 100, 107]

The code:

def powerset(s):
    x = len(s)
    masks = [1 << i for i in range(x)]
    for i in range(1, (1 << x)-1):
        yield [ss for mask, ss in zip(masks, s) if i & mask]

my_Subsets = list(powerset(L)) # <--- THIS TAKES WAY TOO LONG

Answer 1

Your set has 55 elements. Meaning 2^55=36028797018963968 subsets.

There's no way, in any language, any algorithm to make that fast. Because for each subset you need at least one allocation, and that single operation repeated 2^55 times will run forever. For example if we were to run one allocation per nanosecond (in reality this is orders of magnitude slower) we are looking at something over a year (if my calculations are correct). In Python probably 100 years. :P

Not to mention that the final result is unlikely to fit in the entire world's data storage (ram + hard drives) currently available. And definitely not in a single machine's storage. And so final list(...) conversion will fail with 100% probability, even if you wait those years.

Whatever you are trying to achieve (this is likely an XY problem) you are doing it the wrong way.

Answer 2

What you could do is create a class that will behave like a list but would only compute the items as needed and not actually store them:

class Powerset:

    def __init__(self,base):
        self.base = base

    def __len__(self):
        return 2**len(self.base)-2 # - 2 you're excluding empty and full sets

    def __getitem__(self,index):
        if isinstance(index,slice):
            return [ self.__getitem__(i) for i in range(len(self))[index] ]
        else:
            return [ss for bit,ss in enumerate(self.base) if (1<<bit) & (index+1)]


L = [0, 3, 5, 6, 8, 9, 11, 13, 16, 18, 19, 20, 23, 25, 28, 29, 30, 32, 33, 35, 36, 38, 42, 43, 44, 45, 49, 50, 51, 53, 54, 56, 57, 62, 63, 64, 65, 66, 67, 71, 76, 78, 79, 81, 82, 84, 86, 87, 90, 92, 96, 97, 98, 100, 107]


P = Powerset(L)

print(len(P)) # 36028797018963966
print(P[:10]) # [[0], [3], [0, 3], [5], [0, 5], [3, 5], [0, 3, 5], [6], [0, 6], [3, 6]]
print(P[3:6]) # [[5], [0, 5], [3, 5]]
print(P[-3:]) # [[5, 6, 8, 9, 11, 13, 16, 18, 19, 20, 23, 25, 28, 29, 30, 32, 33, 35, 36, 38, 42, 43, 44, 45, 49, 50, 51, 53, 54, 56, 57, 62, 63, 64, 65, 66, 67, 71, 76, 78, 79, 81, 82, 84, 86, 87, 90, 92, 96, 97, 98, 100, 107], [0, 5, 6, 8, 9, 11, 13, 16, 18, 19, 20, 23, 25, 28, 29, 30, 32, 33, 35, 36, 38, 42, 43, 44, 45, 49, 50, 51, 53, 54, 56, 57, 62, 63, 64, 65, 66, 67, 71, 76, 78, 79, 81, 82, 84, 86, 87, 90, 92, 96, 97, 98, 100, 107], [3, 5, 6, 8, 9, 11, 13, 16, 18, 19, 20, 23, 25, 28, 29, 30, 32, 33, 35, 36, 38, 42, 43, 44, 45, 49, 50, 51, 53, 54, 56, 57, 62, 63, 64, 65, 66, 67, 71, 76, 78, 79, 81, 82, 84, 86, 87, 90, 92, 96, 97, 98, 100, 107]]

Obviously, if the next thing you do is a sequential search or traversal of the powerset, it will still take forever.