找到两个非常大的列表之间的差异

Question

I have two very large lists, one is 331991 elements long, lets call that one a, and the other is 99171 elements long, call it b. I want to compare a to b and then return a list of elements in a that are not in b. This also needs to be efficient as possible and in the order that they appear, that is probably a given but I thought I may as well throw it in there.

Answer 1

It can be done in O(m + n) time where m and n correspond to the lengths of the two lists:

exclude = set(b)  # O(m)

new_list = [x for x in a if x not in exclude]  # O(n)

The key here is that sets have constant-time containment tests. Perhaps you could consider having b be a set to begin with.

See also: List Comprehension

---

Using your example :

>>> a = ['a','b','c','d','e']
>>> b = ['a','b','c','f','g']
>>> 
>>> exclude = set(b)
>>> new_list = [x for x in a if x not in exclude]
>>> 
>>> new_list
['d', 'e']

Answer 2

Let us assume:

book = ["once", "upon", "time", ...., "end", "of", "very", "long", "story"]
dct = ["alfa", "anaconda", .., "zeta-jones"]

And you want to remove from book list all the items, which are present in dct.

Quick solution:

short_story = [word in book if word not in dct]

Speeding up searches in dct: turn dct into set - this has faster lookups:

dct = set(dct)
short_story = [word in book if word not in dct]

In case, the book is very long and does not fit into memory, you may process it word by word. For this, we may use a generator:

def story_words(fname):
"""fname is name of text file with a story"""
  with open(fname) as f:
    for line in f:
      for word in line.split()
        yield word

#print out shortened story
for word in story_words("alibaba.txt"):
  if word not in dct:
    print word

And in case, also your dictionary would be far too large, you would have to give up speed and iterate also over content of dictionary. But this I skip for now.

Answer 3

Here's one way converting b to a set, then filtering elements from a that are not present:

from itertools import ifilterfalse

a = ['a','b','c','d','e']
b = ['a','b','c']
c = list(ifilterfalse(set(b).__contains__, a))
# ['d', 'e']