扁平嵌套循环/降低复杂性 - 互补对计数算法

Question

I was recently trying to solve some task in Python and I have found the solution that seems to have the complexity of O(n log n) , but I believe it is very inefficient for some inputs (such as first parameter being 0 and pairs being very long list of zeros).

It has also three levels of for loops. I believe it can be optimized, but at the moment I cannot optimize it more, I am probably just missing something obvious ;)

So, basically, the problem is as follows:

Given list of integers ( values ), the function needs to return the number of indexes' pairs that meet the following criteria:

• lets assume single index pair is a tuple like (index1, index2) ,
• then values[index1] == complementary_diff - values[index2] is true,

Example : If given a list like [1, 3, -4, 0, -3, 5] as values and 1 as complementary_diff , the function should return 4 (which is the length of the following list of indexes' pairs: [(0, 3), (2, 5), (3, 0), (5, 2)] ).

This is what I have so far, it should work perfectly most of the time, but - as I said - in some cases it could run very slowly, despite the approximation of its complexity O(n log n) (it looks like pessimistic complexity is O(n^2) ).

def complementary_pairs_number (complementary_diff, values):
    value_key = {} # dictionary storing indexes indexed by values
    for index, item in enumerate(values):
        try:
            value_key[item].append(index)
        except (KeyError,): # the item has not been found in value_key's keys
            value_key[item] = [index]
    key_pairs = set() # key pairs are unique by nature
    for pos_value in value_key: # iterate through keys of value_key dictionary
        sym_value = complementary_diff - pos_value
        if sym_value in value_key: # checks if the symmetric value has been found
            for i1 in value_key[pos_value]: # iterate through pos_values' indexes
                for i2 in value_key[sym_value]: # as above, through sym_values
                    # add indexes' pairs or ignore if already added to the set
                    key_pairs.add((i1, i2))
                    key_pairs.add((i2, i1))
    return len(key_pairs)

For the given example it behaves like that:

>>> complementary_pairs_number(1, [1, 3, -4, 0, -3, 5])
4

If you see how the code could be "flattened" or "simplified", please let me know.

I am not sure if just checking for complementary_diff == 0 etc. is the best approach - if you think it is, please let me know.

EDIT: I have corrected the example (thanks, unutbu!).

Answer 1

I think this improves the complexity to O(n) :

• value_key.setdefault(item,[]).append(index) is faster than using the try..except blocks. It is also faster than using a collections.defaultdict(list) . (I tested this with ipython %timeit.)
• The original code visits every solution twice. For each pos_value in value_key , there is a unique sym_value associated with pos_value . There are solutions when sym_value is also in value_key . But when we iterate over the keys in value_key , pos_value is eventually assigned to the value of sym_value , which make the code repeat the calculation it has already done. So you can cut the work in half if you can stop pos_value from equaling the old sym_value . I implemented that with a seen = set() to keep track of seen sym_value s.

• The code only cares about len(key_pairs) , not the key_pairs themselves. So instead of keeping track of the pairs (with a set ), we can simply keep track of the count (with num_pairs ). So we can replace the two inner for-loops with

   num_pairs += 2*len(value_key[pos_value])*len(value_key[sym_value]) 

  or half that in the "unique diagonal" case, pos_value == sym_value .

---

def complementary_pairs_number(complementary_diff, values):
    value_key = {} # dictionary storing indexes indexed by values
    for index, item in enumerate(values):
        value_key.setdefault(item,[]).append(index)
    # print(value_key)
    num_pairs = 0
    seen = set()
    for pos_value in value_key: 
        if pos_value in seen: continue
        sym_value = complementary_diff - pos_value
        seen.add(sym_value)
        if sym_value in value_key: 
            # print(pos_value, sym_value, value_key[pos_value],value_key[sym_value])
            n = len(value_key[pos_value])*len(value_key[sym_value])
            if pos_value == sym_value:
                num_pairs += n
            else:
                num_pairs += 2*n
    return num_pairs

Answer 2

You may want to look into functional programming idioms, such as reduce, etc.

Often times, nested array logic can be simplified by using functions like reduce, map, reject, etc.

For an example (in javascript) check out underscore js. I'm not terribly smart at Python, so I don't know which libraries they have available.

Answer 3

I think (some or all of) these would help, but I'm not sure how I would prove it yet.

1) Take values and reduce it to a distinct set of values, recording the count of each element (O(n))

2) Sort the resulting array. (n log n)

3) If you can allocate lots of memory, I guess you might be able to populate a sparse array with the values - so if the range of values is -100 : +100, allocate an array of [201] and any value that exists in the reduced set pops a one at the value index in the large sparse array.

4) Any value that you want to check if it meets your condition now has to look at the index in the sparse array according to the x - y relationship and see if a value exists there.

5) as unutbu pointed out, it's trivially symmetric, so if {a,b} is a pair, so is {b,a}.

Answer 4

I think you can improve this by separating out the algebra part from the search and using smarter data structures.

1. Go through the list and subtract from the complementary diff for each item in the list.

    resultlist[index] = complementary_diff - originallist[index] 

   You can use either a map or a simple loop. -> Takes of O(n) time.

2. See if the number in the resulting list exists in the original list.

   • Here, with a naive list, you would actually get O(n^2) , because you can end up searching for the whole original list per item in the resulting list.

   • However, there are smarter ways to organize your data than this. If you have the original list sorted , your search time reduces to O(nlogn + nlogn) = O(nlogn) , nlogn for the sort, and nlogn for the binary search per element.

   • If you wanted to be even smarter you can make your list in to a dictionary(or hash table) and then this step becomes O(n + n) = O(n) , n to build the dictionary and 1 * n to search each element in the dictionary. (*EDIT: * Since you cannot assume uniqueness of each value in the original list. You might want to keep count of how many times each value appears in the original list.)

So with this now you get O(n) total runtime.

Using your example:

1, [1, 3, -4, 0, -3, 5],

1. Generate the result list:

    >>> resultlist [0, -2, 5, 1, 4, -4]. 

2. Now we search:

   • Flatten out the original list into a dictionary. I chose to use the original list's index as the value as that seems like a side data you're interested in.

      >>> original_table {(1,0), (3,1), (-4,2), (0,3), (-3,4), (5,5)} 

   • For each element in the result list, search in the hash table and make the tuple:

      (resultlist_index, original_table[resultlist[resultlist_index]]) 

     This should look like the example solution you had.

3. Now you just find the length of the resulting list of tuples.

Now here's the code:

example_diff = 1
example_values = [1, 3, -4, 0, -3, 5]
example2_diff = 1
example2_values = [1, 0, 1]

def complementary_pairs_number(complementary_diff, values):
    """
        Given an integer complement and a list of values count how many pairs
        of complementary pairs there are in the list.
    """
    print "Input:", complementary_diff, values
    # Step 1. Result list
    resultlist = [complementary_diff - value for value in values]
    print "Result List:", resultlist

    # Step 2. Flatten into dictionary
    original_table = {}
    for original_index in xrange(len(values)):
        if values[original_index] in original_table:
            original_table[values[original_index]].append(original_index)
        else:
            original_table[values[original_index]] = [original_index]
    print "Flattened dictionary:", original_table

    # Step 2.5 Search through dictionary and count up the resulting pairs.
    pair_count = 0
    for resultlist_index in xrange(len(resultlist)):
        if resultlist[resultlist_index] in original_table:
            pair_count += len(original_table[resultlist[resultlist_index]])
    print "Complementary Pair Count:", pair_count

    # (Optional) Step 2.5 Search through dictionary and create complementary pairs. Adds O(n^2) complexity.
    pairs = []
    for resultlist_index in xrange(len(resultlist)):
        if resultlist[resultlist_index] in original_table:
            pairs += [(resultlist_index, original_index) for original_index in
                original_table[resultlist[resultlist_index]]]
    print "Complementary Pair Indices:", pairs

    # Step 3
    return pair_count

if __name__ == "__main__":
    complementary_pairs_number(example_diff, example_values)
    complementary_pairs_number(example2_diff, example2_values)

Output:

$ python complementary.py
Input: 1 [1, 3, -4, 0, -3, 5]
Result List: [0, -2, 5, 1, 4, -4]
Flattened dictionary: {0: 3, 1: 0, 3: 1, 5: 5, -4: 2, -3: 4}
Complementary Pair Indices: [(0, 3), (2, 5), (3, 0), (5, 2)]
Input: 1 [1, 0, 1]
Result List: [0, 1, 0]
Flattened dictionary: {0: [1], 1: [0, 2]}
Complementary Pair Count: 4
Complementary Pair Indices: [(0, 1), (1, 0), (1, 2), (2, 1)]

Thanks!

Answer 5

Modified the solution provided by @unutbu:

The problem can be reduced to comparing these 2 dictionaries:

1. values

2. pre-computed dictionary for (complementary_diff - values[i])

    def complementary_pairs_number(complementary_diff, values): value_key = {} # dictionary storing indexes indexed by values for index, item in enumerate(values): value_key.setdefault(item,[]).append(index) answer_key = {} # dictionary storing indexes indexed by (complementary_diff - values) for index, item in enumerate(values): answer_key.setdefault((complementary_diff-item),[]).append(index) num_pairs = 0 print(value_key) print(answer_key) for pos_value in value_key: if pos_value in answer_key: num_pairs+=len(value_key[pos_value])*len(answer_key[pos_value]) return num_pairs