Python：仅在列表末尾删除重复的值

Question

I have a python list where order of responses is important. I would like to filter out nan values only if they occur at the end of the list. I was wondering if there is an efficient way to go from a list like the following:

nan = float("nan")
responses = [1.0, nan, 9.0, nan, nan, nan, nan, nan, nan, nan, nan]

To a list without any trailing nan values:

[1.0, nan, 9.0]

I know how to filter out all nan values using a list comprehension:

import pandas as pd
[r for r in responses if pd.notnull(r)]
>>> [1.0, 9.0]

But can't think of a straightforward way to filter out nan values at the end without converting everything to strings and using regular expressions. I could do that, but am concerned about performance, which is an issue because it will be performed several hundred thousand times.

Answer 1

while responses and math.isnan(responses[-1]):
    responses.pop()

Update: this isn't as fast as a straight up slice.

>>> timeit.timeit('responses = list(r)\nwhile responses and isnan(responses[-1]): responses.pop()', 'from math import isnan; nan = float("nan"); r = [1.0, nan, 9.0, nan, nan, nan, nan, nan, nan, nan, nan]')
1.3209394318982959
>>> timeit.timeit('responses = list(r)\nresponses = responses[:3]', 'from math import isnan; nan = float("nan"); r = [1.0, nan, 9.0, nan, nan, nan, nan, nan, nan, nan, nan]')
0.29652016144245863

Answer 2

There is no builtin function or method. But you can use a loop:

while responses and math.isnan(responses[-1]):
    del responses[-1]

As you can see yourself, this runs in linear time and uses no extra space.

Answer 3

You can reverse it and use itertools.dropwhile . This should work for any value.

r = [1.0, nan, 9.0, nan, nan, nan, nan, nan, nan, nan, nan]
list(itertools.dropwhile(lambda x: x == r[-1], reversed(r)))[::-1] + r[-1:]

To filter just nan , you can replace lambda x: x == r[-1] for math.isnan :

list(itertools.dropwhile(math.isnan, reversed(r)))[::-1]

Answer 4

What I would do is iterate over the list once, and then find where the end sequence of nans begins. Something like

responses = [1.0, 'nan', 9.0, 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan']

first_index = -1
for i, val in enumerate(responses):
  if val == 'nan':
    if first_index == -1:
      first_index = i
  else:
    first_index = -1

responses = responses[:first_index]  # [1.0, 'nan', 9.0]

Then you can perform a single slice operation. It's a bit more verbose, than other solutions, but should be quicker.

Time Complexity

According to this page , the slice operation is O(n), and iterating over the list is O(n), making this entire algorithm O(n) complexity.

Even better would be to iterate over the list backwards.