比较两个大列表以查找满足python中条件的元素对

Question

I have two large lists of numbers (maybe a million elements each). I would like to compare both of them element-wise to identify element pairs that have a difference of less than 0.5. I know two nested for loops is not an option. Is there any quick way to do this using sets or zip?

For eg. if my lists are list1 = [1,2,3,4] and list2 = [3,4,5,6] and the condition is difference of 1, then the solution would have the pairs arranged in a list [element from list1 , element from list2 , difference]. The solution would be [[2,3,1],[3,3,0],[3,4,1],[4,3,1],[4,4,0],[4,5,1]]

Thanks

Answer 1

This should work. (Criticism appreciated)

Basically, my idea is to sort the two list O(nlogn), and then go through the list , keeping in memory the distance with the next element, and therefore, not computing all the pairs, but only a subset giving me a O(2*m*n) m being the maximum distance allowed

x = sorted([0, 2, 3, 4])
y = sorted([1,3, 4, 5, 6])
index = 0
delta = 1
output = []
j = 0
value_2 = y[0]
no_more = False
number_of_operation = 0
for i,value_1 in enumerate(x[:]):
    print(f'Testing for this {value_1}')
    skip = False
    try:
        next_value_at = x[i+1] - value_1 
        if next_value_at > delta:
            skip = True
            print('We can directly skip to next')
    except:
        print('At the end of list')
    while value_2 - value_1 <= delta:
        number_of_operation+=1
        print(value_1,value_2)
        try:
            if abs(value_1 - value_2) <= delta:
                output += [[value_1,value_2,value_1-value_2]]
            j+=1
            value_2 = y[j]
            print(value_1,value_2) 
            continue
        except:
            no_more = True
            print('end of list')
            break
    if not skip:
        print("Going back")
        j=index
        value_2 = y[index]
    else:
        index = j
    if no_more:
        print('end')
        break
    print(number_of_operation)

Answer 2

Use numpy's broadcasting

import numpy as np
x = np.array([1, 2, 3, 4]).reshape(-1, 1)
y = np.array([3, 4, 5, 6]).reshape(1, -1)
diff = x - y

However, you can't avoid N^2 comparisons, only take advantage of numpy's speed optimizations.

Answer 3

You might be able to avoid the O(N²) behavior if you sort your lists first (or better yet if your lists are already sorted). Then you can step through them element wise. This would give you O(nLogn) for the sorts plus O(n) to step through the elements. For example:

list1 = range(0, 1000000)
list2 = range(999999, 1999999)

def getClose(list1, list2):
    c1, c2 = 0, 0
    while c1 < len(list1) and c2 < len(list2):
        if abs(list1[c1] - list2[c2]) <= 1:
            yield (list1[c1], list2[c2], abs(list1[c1] - list2[c2]))
        if list1[c1] < list2[c2]:
            c1 += 1
        else:
            c2 += 1

for n in getClose(list1, list2):
    print(n)

Produces...

999998 999999 1
999999 999999 0
999999 1000000 1

...relatively quickly and much quicker than finding the product first.