Fastest way to apply function on different lists in python

Question

I have complex algorithm to build in order to select the best combination of elements in my list.

I have a list of 20 elements. I make all the combinations this list using this algorithms, the resutlt would be a list of element with size: 2^20-1 (without duplications)

from itertools import combinations

def get_all_combinations(input_list):
    for i in xrange(len(input_list)):
        for item in combinations(input_list, r = i + 1):
            yield list(item)    

input_list = [1,4,6,8,11,13,5,98,45,10,21,34,46,85,311,133,35,938,345,310]
print  len(get_all_combinations(input_list)) # 1048575

I have another algorithm that is applied on every list, then calculate the max.

// this is just an example
def calcul_factor(item):
 return max(item) * min(item) / sqrt(min(item))

I tried to do it like this way: but it's taking a long time.

columnsList= get_all_combinations(input_list)
for x in columnsList:
  i= calcul_factor(x)
  factorsList.append(i)
  l.append(x)


print "max", max(factorsList)
print "Best combinations:", l[factorsList.index( max(factorsList))] 

Does using Maps/Lamda expressions solve issues to make "parallelisme" to calculate the maximum ? ANy hints to do that ?

Answer 1

In case you can't find a better algorithm (which might be needed here) you can avoid creating those big lists by using generators. With the help of itertools.chain you can combine the itertools.combinations -generators. Furthermore the max -function can take a function as a key. Your code can be reduced to:

all_combinations = chain(*[combinations(input_list, i) for i in range(1, len(input_list))])
max(all_combinations, key=algorithm)

Since this code relies solely on generators it might be faster (doesn't mean fast enough).

Edit: I generally agree with Hugh Bothwell, that you should be trying to find a better algorithm before going with an implementation like this. Especially if your lists are going to contain more than 20 elements.

Answer 2

If you can easily calculate calcul_factor(item + [k]) given calcul_factor(item) , you might greatly benefit from a dynamic-programming approach.

If you can eliminate some bad solutions early, it will also greatly reduce the total number of combinations to consider (branch-and-bound).

If the calculation is reasonably well-behaved, you might even be able to use ie simplex method or a linear solver and walk directly to a solution (something like O(n**2 log n) runtime instead of O(2**n))

Could you show us the actual calcul_factor code and an actual input_list ?