try-except block in for loop skips NEXT iteration?

Question

I have text data that I am trying to clean for numerical values. I break it into as clean rows as I can, and separate the rows into data points. An example is

["1.115","","","4.3"]

My code should turn that into

["1.115","4.3"]

Here is the snippet:

for i in t:
    try:
        print(float(i))
    except ValueError:
        print(i)
        t.remove(i)
        continue

All the print() statements are there for debugging. Running the code gives

["1.115","","4.3"]

As output. If there are no two non-floats in a row, it runs fine, but after removing a non-float through the exception handling, it does not print the next float value.

Answer 1

This looks like an issue with modifying a list that you're currently looping over -- by removing an element you've changed the meaning of the cureent offset. One fix is to create a new list instead of changing the original one:

t = ["1.115", "", "", "4.3"]

s = []

for i in t:
    try:
        s.append(float(i))
    except ValueError:
        pass

print(s)

If you really want to the loop to modify the original list, you can try something like this:

t = ["1.115", "", "", "4.3"]

i = 0

while i < len(t):
    try:
        float(t[i])
        i += 1
    except ValueError:
        del t[i]

print(t)

But make sure you've accounted for all possible cases and test it thoroughly.

Answer 2

Python (and most languages) have a good chance of getting confused if you're modifying a list while holding an iterator to the list.

This code falls prey to that problem, in that you're modifying the same list you're iterating through. It's more common to construct a new list that is the output. Here's an example:

def yield_only_floats(l):
    for s in l:
        try:
            float(s)
            yield s
        except ValueError:
            pass

x = list(yield_only_floats(["1.115","","","4.3"]))

print x

Gets you a result of ['1.115', '4.3']

If you want to modify an original list, you can still do it:

x = ["1.115","","","4.3"]
x[:] = list(yield_only_floats(x))

But, if you really want to modify the same list you are iterating while iterating, the best idea is to iterate in reverse:

def leave_only_floats(l):
    for i in xrange(len(l) - 1, -1, -1):
        try:
            float(l[i])
        except ValueError:
            del l[i]

x = ["1.115","","","4.3"]
leave_only_floats(x)

Note that I've also used positional deletion, rather than value deletion, which makes it faster (the list doesn't have to get searched another time)

BTW, you could also consider using a list comprehension:

def is_float(s):
    try:
        float(s)
        return True
    except ValueError:
        return False

x = ["1.115","","","4.3"]
y = [s for s in x if is_float(s)]

Personally, I think the list comprehension method is the most readable, for this kind of problem.

Answer 3

It's not a good idea to change object you are iterating over. The list iteration is done index by index, so when you erase one element, remaining elements on the right are shifted down.

t = ["1.115", "", "", "4.3"]

for i in t:
    try:
        print(float(i))
    except ValueError:
        print(i)
        t.remove(i)

# First run of loop: 
idx = 0
i = "1.115"
t = ["1.115", "", "", "4.3"]
# Second run of loop
idx = 1
i = ""
t = ["1.115", "", "4.3"]
# Third, last run of loop
idx = 2
i = "4.3"
t = ["1.115", "", "4.3"]

The right way of doing this:

t = ["1.115","","","4.3"]

def is_float(number):
    try:
        float(number)
        return True
    except ValueError:
        return False

res = [x for x in t if is_float(x)]

Answer 4

Just add .copy() to your loop and remove 'continue' statement:

for i in t.copy():
    try:
        print(float(i))
    except ValueError:
        print(i)
        t.remove(i)