How to continue a loop after catching exception in try … except

Question

I am reading a big file in chunks and I am doing some operations on each of the chunks. While reading one of get them I had the following message error:

pandas.errors.ParserError: Error tokenizing data. C error: Expected 26 fields in line 15929977, saw 118

which means that one of my file lines doesn't follow the same format as the others. What I thought I could do was to just omit this chunk but I couldn't get a way to do it. I tried to do a try/except block as follows:

data = pd.read_table('ny_data_file.txt', sep=',', 
                      header=0, encoding = 'latin1', chunksize = 5000)
try: 
    for chunk in data:
           # operations
except pandas.errors.ParseError:
           # Here is my problem

What I have written here is my problem is that if the chunk is not well parsed, my code will automatically go to the exception not even entering the for loop , but what I would like is to skip this chunk and move forward to the next one, on which I would like to perform the operations inside the loop.

I have checked on stackoverflow but I couldn't find anything similar where the try was performed on the for loop. Any help would be appreciated.

UPDATE:

I have tried to do as suggested in the comments:

try:
    for chunk in data:
        #operations
except pandas.errors.ParserError:
        # continue/pass/handle error

But still is not cathching the exception because as said the exception is created when getting the chyunk out of my data not when doing operations with it.

Answer 1

I understood that, in the operations part you get exception. If it is like that: you should just continue:

for chunk in data:
    try:
       # operations
    except pandas.errors.ParseError:
       # continue

Answer 2

The way you use try - except makes it skip the entire for loop if an exception is caught in it. If you want to only skip one iteration you need to write the try-except inside the loop like so:

for chunk in data:
    try:
       # operations
    except pandas.errors.ParseError as e:
        # inform the user of the error
        print("Error encountered while parsing chunk {}".format(chunk))
        print(e)

Answer 3

I am not sure where the exception is thrown. Maybe adding a full error stack would help. If the error is thrown by the read_table() call maybe you could try this:

try: 
    data = pd.read_table('ny_data_file.txt', sep=',', 
                      header=0, encoding = 'latin1', chunksize = 5000)

except pandas.errors.ParseError:
           pass
for chunk in data:
           # operations

Answer 4

正如@JonClements什么解决我的问题建议是使用error_bad_lines=False在pd.read_csv所以它只是跳过该行造成的麻烦，让我执行的，其余为循环。