Pandas.read_csv() 忽略包含 FEWER 字段的坏行/行。 文本文件

Question

I am trying to read this huge text file: https://www.dropbox.com/s/3ikikw8bxde6y1i/TCAD_SPECIAL%20EXPORT_2019_20200409.zip?dl=0 (if you download the zip, the file is Special_ARB.txt (not necessary for my question imo).

I am running this code (adding error_bad_lines=False ) to ignore lines with more-than-expected fields, which works well:

pd.read_csv(r'~/Special_ARB.txt', sep="|", 
            header=None,encoding='cp1252',error_bad_lines=False)

The problem is that read.csv() crashed when a line had only 1 field. With the following error:

Too many columns specified: expected 77 and found 1

Is there a way to tell python/pandas to ignore this error? It is not letting me know which line it is. There are more than a million rows so I can't just find it on my own.

• I tried a for loop to read line by line and figure it out from there, but data is so large that python crashed.
• The number of columns is 77 which is correctly identify by pandas when running the code, I don't think that's an issue.

Thanks,

Answer 1

• Errors and Exceptions
• Python Try Except

try:
   pd.read_csv(r'~/Special_ARB.txt', sep="|", header=None,encoding='cp1252',error_bad_lines=False)
except <your error description>:
   <do this>

Answer 2

This should work for in-memory datasets, you can use chunking for a solution on large datasets: https://stackoverflow.com/a/59331754/9379924