在Python中使用二进制搜索比较列表列表

Question

I have 2 list of lists x(1 million elements) and y(0.1 million elements) and want to get z=xy. each list consist of sub lists of 4 elements each, of which the first element of each sublist is sorted. The first element are strictly increasing and no duplicates are present. Now I did this using list comprehension and it roughly takes 6.5 hrs to run it. I wanted to know what is the most time efficient way to do this, keeping in mind that my end result should also be a list of lists.

Secondly, since all my first elements are sorted I thought doing a binary search would be a better idea. Idea of binary search - for ex consider I have 2 lists of size x=30 and y=10 I am looping over elements of y and comparing the first element of each sub list to that of the elemnts in x using binary search, when I find a match that sublist is deleted from the x list. So the expected output list should contain 20 elements.But the code I have written gives me 23(it does not delete the last three matches) and I dont know whats wrong with it. Heres the code:

def intersection(x,y):
    temp=x[:]
    for i in range(len(y)):
        l=0
        h=len(x)-1
        while l<h:
            mid=l+((h-l)/2)
            if y[i][0]==temp[mid][0]:
                a=y[i]
                x.remove(a)
                break
            elif y[i][0]>temp[mid][0]:
                if l==mid:
                    break
                l=mid
            elif y[i][0]<temp[mid][0]:
                h=mid
    return(x)






X-List input of 30 elements
[[1.0, 25.0, 0.0, 0.0]
[2.0, 0.0, 25.0, 0.0]
[3.0, 0.0, 50.0, 0.0]
[4.0, 50.0, 50.0, 0.0]
[5.0, 50.0, 0.0, 0.0]
[6.0, 0.0, 25.0, 10.0]
[7.0, 25.0, 0.0, 10.0]
[8.0, 50.0, 0.0, 10.0]
[9.0, 50.0, 50.0, 10.0]
[10.0, 0.0, 50.0, 10.0]
[11.0, 0.0, 0.0, 0.0]
[12.0, 0.0, 0.0, 10.0]
[13.0, 17.6776695, 17.6776695, 0.0]
[14.0, 0.0, 34.3113632, 0.0]
[15.0, 25.9780293, 50.0, 0.0]
[16.0, 50.0, 25.9780293, 0.0]
[17.0, 34.3113632, 0.0, 0.0]
[18.0, 17.6776695, 17.6776695, 10.0]
[19.0, 34.3113632, 0.0, 10.0]
[20.0, 50.0, 25.9780293, 10.0]
[21.0, 25.9780293, 50.0, 10.0]
[22.0, 0.0, 34.3113632, 10.0]
[23.0, 11.6599302, 0.0, 0.0]
[24.0, 0.0, 11.6599302, 0.0]
[25.0, 0.0, 11.6599302, 10.0]
[26.0, 11.6599302, 0.0, 10.0]
[27.0, 27.9121876, 27.9121876, 0.0]
[28.0, 27.9121876, 27.9121876, 10.0]
[29.0, 9.77920055, 9.77920055, 0.0]
[30.0, 9.77920055, 9.77920055, 10.0]]
Y -List of 10 elements
[1.0, 25.0, 0.0, 0.0]
[2.0, 0.0, 25.0, 0.0]
[11.0, 0.0, 0.0, 0.0]
[13.0, 17.6776695, 17.6776695, 0.0]
[14.0, 0.0, 34.3113632, 0.0]
[17.0, 34.3113632, 0.0, 0.0]
[23.0, 11.6599302, 0.0, 0.0]
[24.0, 0.0, 11.6599302, 0.0]
[27.0, 27.9121876, 27.9121876, 0.0]
[29.0, 9.77920055, 9.77920055, 0.0]
------------------------------------------------------------------------------------------------------------------------------------------Z list (X-Y) the result should be 20 elements but its gives length as 23 elements. it does not remove the remaining 3 elements from the list.




[[3.0, 0.0, 50.0, 0.0],
 [4.0, 50.0, 50.0, 0.0],
 [5.0, 50.0, 0.0, 0.0],
 [6.0, 0.0, 25.0, 10.0],
 [7.0, 25.0, 0.0, 10.0],
 [8.0, 50.0, 0.0, 10.0],
 [9.0, 50.0, 50.0, 10.0],
 [10.0, 0.0, 50.0, 10.0],
 [12.0, 0.0, 0.0, 10.0],
 [15.0, 25.9780293, 50.0, 0.0],
 [16.0, 50.0, 25.9780293, 0.0],
 [18.0, 17.6776695, 17.6776695, 10.0],
 [19.0, 34.3113632, 0.0, 10.0],
 [20.0, 50.0, 25.9780293, 10.0],
 [21.0, 25.9780293, 50.0, 10.0],
 [22.0, 0.0, 34.3113632, 10.0],
 [24.0, 0.0, 11.6599302, 0.0],
 [25.0, 0.0, 11.6599302, 10.0],
 [26.0, 11.6599302, 0.0, 10.0],
 [27.0, 27.9121876, 27.9121876, 0.0],
 [28.0, 27.9121876, 27.9121876, 10.0],
 [29.0, 9.77920055, 9.77920055, 0.0],
 [30.0, 9.77920055, 9.77920055, 10.0]]

Answer 1

If I understand you correctly, use bisect.bisect_left to find the the matches and delete:

from bisect import bisect_left

for ele in y:
    ind = bisect_left(x, ele)
    if ind < len(x) -1 and x[ind][0] == ele[0]:
        del x[ind]

If you look at the source you can see the code used for bisect_left:

def bisect_left(a, x, lo=0, hi=None):
    """Return the index where to insert item x in list a, assuming a is sorted.

    The return value i is such that all e in a[:i] have e < x, and all e in
    a[i:] have e >= x.  So if x already appears in the list, a.insert(x) will
    insert just before the leftmost x already there.

    Optional args lo (default 0) and hi (default len(a)) bound the
    slice of a to be searched.
    """

    if lo < 0:
        raise ValueError('lo must be non-negative')
    if hi is None:
        hi = len(a)
    while lo < hi:
        mid = (lo+hi)//2
        if a[mid] < x: lo = mid+1
        else: hi = mid
    return lo

You can adapt that into your own code:

def intersection(x, y):
    for ele in y:
        lo = 0
        hi = len(x)
        while lo < hi:
            mid = (lo+hi)//2
            if x[mid] < ele:
                lo = mid+1
            else:
                hi = mid
        if lo < len(x) - 1 and x[ind][0] == ele[0]:
            del x[lo]
    return x

print(len(intersection(x,y)))
20

If you have dupes then you will need to use remove. Checking the first elements for an exact match is if lo < len(x) - 1 and x[ind][0] == ele[0]: but if you were using remove I don't see how that could work, just because the first elements matched does not mean y[i] was in x so x.remove would fail. So if you are only matching first elements then you can you can change your logic and just iterate over x putting all first elements from each sublist in a set and using a generator expression to update x.

st = {sub[0] for sub in y}

x[:] = (sub for sub in x if sub[0] not in st)

Answer 2

Bisection can work, but another easy solution is to use a set :

y_set = set(tuple(v) for v in y)

Note that the list s have to be turned into something immutable.

Now simply generate the result:

z = [v for v in x if tuple(v) not in y_set]

This might look very similar to your initial solution, but the lookups here are much faster.

@StefanPochmann has a good point that you might want to base your lookup on something more specific than the whole vector, such as just the first element. The question wasn't very clear about that (only stating those are sorted).

Answer 3

If you can use the first elements for filtering:

ykeys = set(zip(*y)[0])
z = [s for s in x if s[0] not in ykeys]

Python 3 versions:

ykeys = set(list(zip(*y))[0])
ykeys = {s[0] for s in y}

If judging by the first element alone is not enough:

yset = set(map(tuple, y))
return [s for s in x if tuple(s) not in yset]

On my weak laptop, with a test of your size, the first solution takes about 0.4 seconds and the second solution takes about 1 second. Not that surprising, since set lookups average O(1) ).

Here's a third version, and this one might be the most interesting because it doesn't just let Python do the job and because it's closer to what you intended but even better:

yi, last = 0, len(y) - 1
z = []
for s in x:
    while s > y[yi] and yi < last:
        yi += 1
    if s != y[yi]:
        z.append(s)

This walks over x , and "in parallel" walks over y . Similar to the merge step of merge-sort. With yi we point into y , and we increase it as needed. Thus we have overall linear time, as we only walk over x from start to end and also over y from start up to end. My laptop takes about 0.6 seconds for this, which is faster than my second solution! (It's not fair to compare it to my first solution, since that one only looks at the first elements).