Python中的算法帮助，找到对（x，y），其中y / x> const

Question

I'm building a rather huge real-time odds system, and my bottleneck right now is the actual computation. I have a huge amount of sorted lists, and for each list, I need to find each pair (x,y) where (y/x) > const.

This is what I'm currently doing;

for f in reversed(xrange(1, len(odds))):
    found = False
    for s in xrange(0, f):
        try:
            edge = odds[s]/odds[f]
        except ZeroDivisionError:
            continue
        if edge > const:
            found = True
            yield odds[f], odds[s]
        else:
            break
    if not found:
        break

The plan being stop whenever I'm certain there are no more pairs. However, I'm doing this for an average of 40 lists each cycle, and I'm in desperate need of shortening the cycletime. I'm curious about using numpy and see whether than can help me.

The length of each inidividual list is < 50.

Thanks for any help!

EDIT This is an examplelist with structure

(_ , odds1, odds2, odds3, _, _) (_ means not used):
[(260, Decimal('1.45'), Decimal('5.5'), Decimal('4'), 0, 2666298), (35549, Decimal('1.62'), Decimal('4.5'), Decimal('3.5'), 0, 2666298), (35551, Decimal('1.666'), Decimal('4.333'), Decimal('3.6'), 0, 2666298), (35552, Decimal('1.6'), Decimal('3.6'), Decimal('3.35'), 0, 2666298), (35553, Decimal('1.6'), Decimal('3.6'), Decimal('3.35'), 0, 2666298), (54453, Decimal('1.65'), Decimal('4.2'), Decimal('3.6'), 0, 2666298), (56234, Decimal('1.571'), Decimal('4.65'), Decimal('3.9'), 0, 2666298), (56911, Decimal('1.7'), Decimal('4.2'), Decimal('3.15'), 0, 2666298)]

I split this list into 3 lists, odds1_list, odds2_list, odds3_list and do computations on them. An example of odds1:

[Decimal('1.7'), Decimal('1.666'), Decimal('1.65'), Decimal('1.62'), Decimal('1.6'), Decimal('1.6'), Decimal('1.571'), Decimal('1.45')]

Then I need to identify all pairs (x,y) in this list where (y/x > const)

Answer 1

If you have some list odds you can do

from itertools import product
list(filter(lambda i: i[0] != 0 and i[1]/i[0] > 2, product(odds,repeat=2)))

For example

odds = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

Produces

[(1.0, 3.0), (1.0, 4.0), (1.0, 5.0), (1.0, 6.0), (1.0, 7.0), (1.0, 8.0), (1.0, 9.0),
 (2.0, 5.0), (2.0, 6.0), (2.0, 7.0), (2.0, 8.0), (2.0, 9.0),
 (3.0, 7.0), (3.0, 8.0), (3.0, 9.0),
 (4.0, 9.0)]

Answer 2

If the list is sorted, then for each x you can just search the list for the first occurrence of const*x, and all items after that match:

import numpy

odds = numpy.arange(10.)
const = 2.5

for x in odds:
    idx = numpy.searchsorted(odds, const*x, side='right')
    for y in odds[idx:]:
        print (x,y)

Running gives

(0.0, 1.0)
(0.0, 2.0)
(0.0, 3.0)
(0.0, 4.0)
(0.0, 5.0)
(0.0, 6.0)
(0.0, 7.0)
(0.0, 8.0)
(0.0, 9.0)
(1.0, 3.0)
(1.0, 4.0)
(1.0, 5.0)
(1.0, 6.0)
(1.0, 7.0)
(1.0, 8.0)
(1.0, 9.0)
(2.0, 6.0)
(2.0, 7.0)
(2.0, 8.0)
(2.0, 9.0)
(3.0, 8.0)
(3.0, 9.0)

Answer 3

If I get you right: Having a list[start, end), you want to find all indices y where list[y] > constant * list[x] for each index x in the sorted list of numbers.

An algorithm might be:

Set the index y to the beginning of the list.
For each index x:
     Set limit := constant * list[x]
     Binary search an index y' in the range [y, end) where list[y'] > limit
     If the index y' is in the range [y, end):
         Add all pairs list[x], list[y''] where y'' is in the range [y', end]
            to the result set.
         Set y = y'
     Otherwise:
         No further results exist.

An implementation in c++ (you accidental tagged it that way):

#include <iostream>
#include <vector>
#include <algorithm>

int main ()
{
    const unsigned constant = 2;
    std::vector<unsigned> v = { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
    auto y = v.begin();
    for(auto x = v.begin(); x < v.end() && y != v.end(); ++x) {
        std::cout << "x = " << *x << ", y: ";
        unsigned limit = constant * (*x);
        y = std::lower_bound(y, v.end(), limit);
        if(y != v.end()) {
            if(*y == limit) ++y;
            for(auto r = y; r < v.end(); ++r)
                std::cout << *r << " ";
        }
        std::cout << "\n";
    }
}

Answer 4

Here's an alternative that uses numpy and its broadcasting ability:

def find_pairs(odds, const):
    with np.errstate(divide='ignore'):
        pairs = odds[np.column_stack(np.where(odds / odds[:, None] > const))]
    return pairs

In theory, the time complexity is O(n**2) (where n is the length of odds ), but you say n is at most 50, which is small enough that the theoretical complexity might not matter.

Here's a full script that includes some of the other answers (so far):

from itertools import product
import numpy as np


def find_pairs(odds, const):
    with np.errstate(divide='ignore'):
        pairs = odds[np.column_stack(np.where(odds / odds[:, None] > const))]
    return pairs


def dursi(odds, const):
    for x in odds:
        idx = np.searchsorted(odds, const*x, side='right')
        for y in odds[idx:]:
            yield (x,y)


def cyber(odds, const):
    return list(filter(lambda i: i[1]/i[0] > const, product(odds, repeat=2)))

And here's a timing comparison, using a numpy array with 50 elements:

In [122]: const = 1.25

In [118]: odds = np.sort(1 + np.random.rand(50))

In [119]: %timeit find_pairs(odds, const)
10000 loops, best of 3: 34.9 µs per loop

In [120]: %timeit list(dursi(odds, const))
10000 loops, best of 3: 150 µs per loop

In [121]: %timeit cyber(odds, const)
1000 loops, best of 3: 541 µs per loop

In this case, the vectorized calculation in find_pairs gives enough of an advantage over explicit python loops that it is faster than the others.