在python中删除多个列表中的几个项目的最有效方法？

Question

I have several lists of items. There are no duplicates, each item appears at most once per list (and normally, only once in all lists). I also have a list of items to remove from this dataset. How can it be done in the cleanest and most efficient way?

I have read that in python, creating a new object is often simplier and faster than filtering an existant one. But I do not observe that in my basic tests :

data = [[i*j for j in range(1, 1000)] for i in range(1, 1000)]
kill = [1456, 1368, 2200, 36, 850, 9585, 59588, 60325, 9520, 9592, 210, 3]

# Method 1 : 0.1990 seconds
for j in kill:
    for i in data:
        if j in i:
            i.remove(j)

# Method 2 : 0.1920 seconds
for i in data:
    for j in kill:
        if j in i:
            i.remove(j)

# Method 3 : 0.2790 seconds
data = [[j for j in i if j not in kill] for i in data]

Which method is the best to use in Python ?

Answer 1

https://wiki.python.org/moin/TimeComplexity

remove is O(n) because it first searches linearly through the list and then, if it finds it, every element after the removed object gets shifted one position to the left in memory. Because of this remove is quite an expensive operation.

Hence remove M items from a list of length N be comes O(N*M)

in on lists is also O(n) because we need to search through the whole list in order. Hence building a new list with a filter is also O(N*M) . However, in on sets is O(1) due to hashing making our filter O(N)

Hence the best solution is (I'm just going to use a flat list for simplicity, not nested)

def remove_kill_from_data(data, kill):
    s = set(kill)
    return [i for i in data if i not in kill]

If you don't care about keeping the order, this would be even faster (due to being done at the C level, it's still O(N) )

def remove_kill_from_data_unordered(data, kill):
    s = set(kill)
    d = set(data)
    return d - s

Applying to your list of lists

kill_set = set(kill)
[remove_kill_from_data(d, kill_set) for d in data]

---

Some timings (each copies from a static data first)

%timeit method1(data, kill)
210 ms ± 769 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit method2(data, kill)
208 ms ± 2.89 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit method3(data, kill)
272 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit method4(data, kill)  # using remove_kill_from_data
69.6 ms ± 1.33 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit method5(data, kill) # using remove_kill_from_data_unordered
59.5 ms ± 3.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Answer 2

There is no “best way to remove from a list in Python”. If there were, Python would have only one way to do it. There are different best ways for different problems, which is why Python has different ways to do it.

---

Correctness is far more important than speed. Getting the wrong answer quickly is useless. (Otherwise, the fastest solution is to just do nothing at all.) And your first two implementations have two problems.

First, you use remove to find and remove the element by value. Besides being wasteful (you just searched the whole list to find the element, and now you're searching it again to find and remove it), that doesn't do the right thing if there are any duplicates—only the first one will get removed. And if there aren't any duplicates, you probably should be using a set (or an OrderedSet, if there aren't duplicates but order does matter), which would let you write this both simpler and immensely faster.

Second, you're removing from a list while iterating it. This causes you to miss elements. If you delete element 2, that moves all of the subsequent elements up—so the original element 3 is now element 2, but your next time through the loop is checking element 3. So, if you have two killables in a row, the second one will be missed. You can solve this by iterating in reverse, but it makes things more complicated. Or you can iterate a copy while modifying the original, but that makes things more complicated and costs time and space for the copy.

Both of these problems can be fixed, but this raises an important point: the first two versions are much easier to get subtly wrong, as proven by the fact that you got them wrong and didn't even notice it.

And of course fixing these problems may well make the first two versions a bit slower instead of a bit faster.

---

Even if you fix these problems, mutating an object doesn't do the same thing as making a new object. If someone else has a reference to the same list, they will see the changes with the first two versions, but they'll keep the list they expected with the last version. If that someone else is code on another thread that might be iterating the list at the same time you're working on it, things get even more complicated. Sometimes you want the first behavior, sometimes the second. You can add more complexity onto either version to get the opposite effect (eg, assigning a comprehension to a slice of the whole list, instead of just rebinding the name), but usually it's simpler to directly write the one that you want.

---

Plus, the comprehension version can be trivially changed to an iterative version that only does work on demand (just change one or both sets of brackets to parentheses). And it works on any iterable, not just lists. You can often get a huge performance benefit and/or simplification at a higher level by rewriting your algorithm as a chain of iterator transformations so you never need the whole data set in memory. But other times, you can get a huge performance or simplicity benefit from multiple passes, or random-access patterns, so a list is much better. And that will determine which implementation you want for this piece of code.

---

There's also a space difference. The comprehension takes linear temporary space instead of constant, but on the other hand it can leave you with a smaller final result in memory because of the way Python grows and shrinks lists. (If this matters, you need to test it—the language doesn't even guarantee that lists shrink their storage at all; how they do so is up to each implementation.)

---

Finally, we're talking about a pretty small difference. If this matters in your code, the fact that you're ignoring other options that could give a much larger improvement probably matters a bit more. If you can use a list of sets instead of a list of lists, that difference will be huge. If you can't, at least making kill a set speeds things up, and you can definitely do that. Using numpy might give an order of magnitude improvement. Just running the existing code in PyPy instead of CPython might speed it up almost as much as numpy for a lot less work. Or you might want to write a C extension for your inner loop (which qcould just be a matter of putting the same code in a .pyx file and Cythonizing it). If none of those things seems worth the effort for an order of magnitude or better improvement, why is it worth putting the time you've already put into this for a 50% improvement?

---

Putting some actual numbers to this:

• Method 1: 140ms
• Corrected method 1: 193ms
• Method 3: 190ms
• Method 3 in PyPy: 21.6ms
• [i - kill for i in data] where data is a list of sets and kill is a set: 20.6ms
• data[~np.isin(data, kill)] where data is a np.array : 26.6ms

(I also tried the same tests in Python 2.7; method 3 is about 30% slower, and method 4 about 15% slower, while the others are almost identical.)

---

As a side note, you didn't show us how you tested this code, and the tests are also easy to get subtly wrong. Even if you used timeit , you still need to make sure you're running against the original list each time, not repeating the code against the same already-filtered list (which would mean the first rep is testing the right case, and the other 99999 reps are testing a different case where there are no killables).