Python exception handling inside for loop

Question

I am new to Python, and programming in general. While I think this question may be related to my use of exception handling, it might also be due to a general lack of understanding!

for i in range(0, len(dates)):
    try:
        data.append(WUF.getwx(location[j], dates[i])[1])
        continue
    except xml.etree.ElementTree.ParseError:
        #copy last good row of data and use it for the missing day
        fixdata = data[-1] #[1,2,3,4,5,6,7,8,9,10,11]
        fixdata[10] = 'estimated'
        data.append(fixdata)

When I run the code as written, I get 2 "estimated" lines in data . One for the previous date, and one for the date that is being estimated. If I change the fixdata variable to [1, 2, 3, 4, 5, ...] , then only one line (the intended line for the estimated date) is "estimated".

Any idea what I'm missing here? Thanks!

Answer 1

The issue is in the line:

fixdata = data[-1]

That doesn't actually copy the data, it only copies the reference to the data. fixdata ends up pointing to the original element in the list, so when you then do

fixdata[10] = 'estimated'

it changes the original data.

To actually copy the data, try this:

fixdata = data[-1][:]

[:] copies the whole list, which is what I think you're trying to do.

Answer 2

It sounds like you are appending lists to your data list -- creating a 'list of lists'.

That's good -- I'm sure it's what you are intending to do. But in your exception handler, you are actually modifying existing data:

fixdata = data[-1]

This line assigns a reference to the last row in data to fixdata . At this point, fixdata and data[-1] are the same object . So, when you do this:

fixdata[10] = 'estimated'

Then the list you are 'fixing' is still inside the data list, as well as being referenced by the name fixdata . Two names, one object in memory.

When you append it again, with

data.append(fixdata)

You still aren't copying it; Now, you actually have two references to the very same object inside your data list. This is why it looks like you have two 'estimated' lines -- they're one and the same object.

What you really want to do is to make a copy of the last row, rather than just a reference to it. Change your code to this:

for i in range(0, len(dates)):
    try:
        data.append(WUF.getwx(location[j], dates[i])[1])
        continue
    except xml.etree.ElementTree.ParseError:
        #copy last good row of data and use it for the missing day                 
        fixdata = data[-1][:] #[1,2,3,4,5,6,7,8,9,10,11]
        fixdata[10] = 'estimated'                
        data.append(fixdata)

The syntax list[:] is a shorthand for 'each element of this list', and is used as a quick copy operation. Then fixdata will be its own list, and you can change elements of independently of the original.

Answer 3

A couple more comments about your original code sample (outside of the scope of your question)

1. You never actually use the i variable, except to index back into the dates list. One of Python's strengths is its iterators: you can iterate your for loop directly over the items in the list, without having to count them and use an index variable:

    for date in dates: 

2. Your continue statement doesn't actually do anything. continue tells the interpreter to finish this iteration of the for loop immediately, and start the next iteration right away. In this case, there are no statements after the continue statement, so the loop would continue naturally, without being told to.

3. There really isn't a need to build up the fixdata variable in two statements, append it in a third, and then discard it (We're discarding the fixdata variable, not its contents) You could do it more succinctly like this:

    data.append(data[-1][:10] + ['estimated']) 

   In one line, this copies the first 10 elements of the last row into a new list, adds the (very short) list ['estimated'] to the end, and appends it.

4. Your code will run into problems if your very first row throws an exception -- in that case, you won't be able to access data[-1] -- trying to do so will throw another exception, which will propagate up. Better to guard against that like this:

    except xml.etree.ElementTree.ParseError: if data: # do something 

Putting it all together, I would write the code more like this:

for date in dates:
    try:
        data.append(WUF.getwx(location[j], date)[1])
    except xml.etree.ElementTree.ParseError:
        # as long as we've parsed at least one good row
        if data:
            # copy last good row of data and use it for the missing day
            data.append(data[-1][:10] + ['estimated'])