加权元素的笛卡尔积

Question

I have a collection of sets of elements where each element has a value (0..1) attached to it (actual container type doesn't matter). I'm iterating over the cartesian products, ie combinations of elements with one element taken from each set, something like this:

import random
import itertools

stuff = [[random.random() for _ in range(random.randint(2,3))] for _ in range(2)]

for combo in itertools.product(*stuff):
    print sum(combo)  # yield in actual application

Easy enough, but I would like to get combinations with higher summed value first. This doesn't need to be deterministic, it would be enough for me to have a significantly higher chance of getting a high-value combination before a low-value one.

Is there a clever way of doing this without creating all combinations first? Maybe by sorting/shifting the element-sets in a certain way?

Answer 1

There is indeed a better way to do this, by first sorting the collections in descending order, and then iterating such that we select the initial elements of each collection first. Since they were sorted, this ensures we generally get high-value combinations first.

Let us build our intuition in steps, plotting the results along the way. I have found this helps a great deal in understanding the method.

Current method

First, your current method (edited lightly for clarity).

import random
import itertools
import matplotlib.pyplot as plt

list1 = [random.random() for _ in range(50)]
list2 = [random.random() for _ in range(50)]

values = []

for combo in itertools.product(list1, list2):
    values.append(sum(combo))
    print(sum(combo))           # yield in actual application

plt.plot(values)
plt.show()

Resulting in,

That is just all over the place! We can already do better by imposing some sorted structure. Let us explore this next.

Pre-sorting the lists

list1 = [random.random() for _ in range(50)]
list2 = [random.random() for _ in range(50)]

list1.sort(reverse=True)
list2.sort(reverse=True)

for combo in itertools.product(list1, list2):
    print(sum(combo))           # yield in actual application

Which yields,

Look at the structure of that beauty! Can we exploit this to yield the largest elements first?

Exploiting the structure

For this part, we will have to let go of itertools.product , as it is too general for our tastes. A similar function is easily written, and we can exploit the regularity of our data when we do so. What do we know of the peaks in figure 2? Well, since the data is sorted, they must all occur at lower indices. If we imagine the indices to our collections as some higher-dimensional space, what this means is that we need to prefer points close to the origin - at least initially.

The following 2-D figure supports our intuition,

A graph-based walk through our matrix should suffice, making sure we move to a new element every time. Now, the implementation I will provide below does build-up a set of visited nodes, which is not what you want. Luckily, all visited nodes not on the 'frontier' (the currently reachable but unvisited nodes) can be deleted, which should limit space complexity considerably. I leave it up to you to come up with a clever way to do so.

The code,

import random
import itertools
import heapq


def neighbours(node):       # see https://stackoverflow.com/a/45618158/4316405
    for relative_index in itertools.product((0, 1), repeat=len(node)):
        yield tuple(i + i_rel for i, i_rel
                    in zip(node, relative_index))


def product(*args):
    heap = [(0, tuple([0] * len(args)))]    # origin
    seen = set()

    while len(heap) != 0:                   # while not empty
        idx_sum, node = heapq.heappop(heap)

        for neighbour in neighbours(node):
            if neighbour in seen:
                continue

            if any(dim == len(arg) for dim, arg in zip(neighbour, args)):
                continue                    # should not go out-of-bounds

            heapq.heappush(heap, (sum(neighbour), neighbour))

            seen.add(neighbour)

            yield [arg[idx] for arg, idx in zip(args, neighbour)]


list1 = [random.random() for _ in range(50)]
list2 = [random.random() for _ in range(50)]

list1.sort(reverse=True)
list2.sort(reverse=True)

for combo in product(list1, list2):
    print(sum(combo))

The code walks along the frontier, each time selecting the index with the lowest index sum (a heuristic for 'closeness' to the origin). This works quite well, as the following figure shows,

Answer 2

Inspired by N. Wouda's answer I tried yet another approach. When testing their answer I noticed a pattern in the indices resembling n-ary encoding (here for 3 sets):

...
(1,1,0)
(1,1,1)
(0,0,2)
(0,1,2)
(1,0,2) <- !
(1,1,2)
(0,2,0)
(0,2,1)
(1,2,0)
...

Notice that lower numbers increase before higher ones do. So I replicated this pattern in code:

idx = np.zeros((len(args)), dtype=np.int)
while max(idx) < 50:  # TODO stop condition
    yield [arg[i] for arg,i in zip(args,idx)]

    low = np.min(idx)
    imin = np.argwhere(idx == low)
    inxt = np.argwhere(idx == low+1)

    idx[imin[:-1]] = 0  # everything to the left of imin[-1]
    idx[imin[-1]] += 1  # increase the last of the lowest indices
    idx[inxt[inxt > imin[-1]]] = 0  # everything to the right

I took some shortcuts since I was just testing; the results are not too bad. While in the beginning this function outperforms N. Wouda's solution, it becomes worse the longer it goes. I think the "index-wave" is shaped differently, resulting in higher noise for indices further away from the origin.

Interesting!

Edit I thought this is quite interesting, so I visualized the way the indices are iterated over - JFYI :)

Index wavefront N. Wouda

Index wavefront from this answer