Python比较列表列表

Question

I have a list that contains 10**7 lists in the format:

big_list = [[1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [2, 3, 4, 26, 33, 40], [10, 23, 33, 45, 46, 47]]

Every list contains 6 unique ints.

I need to compare every list to another list:

lst = [1, 3, 4, 10, 23, 46]

and return those where list item intersection is less than 3. So newlist would be:

new_list = [[2, 3, 4, 5, 6, 7], [2, 3, 4, 26, 33, 40]]

At the moment I'm using set intersection, but it takes about 30 seconds to run

Answer 1

import numpy as np
biglist = [[1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [2, 3, 4, 26, 33, 40], [10, 23, 33, 45, 46, 47]]
oldlist = [1, 3, 4, 10, 23, 46]

b = np.array(biglist)
b[np.array([(b == x).any(axis=1) for x in oldlist]).sum(axis=0) < 3]

returns

array([[ 2,  3,  4,  5,  6,  7],
       [ 2,  3,  4, 26, 33, 40]])

The creation of the numpy array takes some time, but the last line is about twice as fast as the list comprehension with set intersections (for 1e6 lists).

EDIT: The following line is even faster than my code above and needs less memory:

b[reduce(np.add, ((b == x).any(axis=1).astype(np.int) for x in oldlist)) < 3]

Answer 2

>>> big_list = [[1, 2, 3, 4, 5, 6], [2, 3, 4, 5, 6, 7], [2, 3, 4, 26, 33, 40], [10, 23, 33, 45, 46, 47]]
>>> normal = set([1, 3, 4, 10, 23, 46])
>>> [x for x in big_list if len(set(x).intersection(normal)) < 3]
[[2, 3, 4, 5, 6, 7], [2, 3, 4, 26, 33, 40]]

Answer 3

I think a "fast" solution should depend on the specification of your problem. For example, if your reference list is just as short as [1, 3, 4, 10, 23, 46], by sorting every list, we can immediately see that all the lists that start with a num bigger than 10, eg [11, x, x, ...] will NOT have more than 3 common elements with the reference. That could already saves a lot of comparisons.