<class ‘decimal.ConversionSyntax’> errors when importing data from a CSV file using Python (tried using CSV and Pandas)

Question

I have a CSV file from a small business that represents their records for about 70,000 orders spread out over about 30 years. My goal is to store this data in a Django project with a DecimalField in the model setup to receive the import.

Before trying to store the data in Django I'm first trying to import this data into Python. While most of the records import fine there are about 1 in 1000 that throw off a <class 'decimal.ConversionSyntax'> error during the import. This is causing me to get incorrect data into Python. I need to either fix how it reads the data or fix the problem in the data itself.

1. I have tried both the python csv library and pandas to open the file.
2. I've looked at the csv file in a text editor (SublimeText) and can't find any anomalies with the way the data is stored.
3. I have tried to convert the data to UTF-8 using Microsoft Excel
4. I have attempted to copy the data out of Excel into SublimeText, then back from SublimeText into a new row in Excel (virgin row with no data in it beforehand).

None of this has resolved the issue.

How can I figure out what is wrong with this data and resolve this error?

Answer 1

You can use chardet to detect the encoding.

import chardet
with open('csv_of_text.csv','rb') as fraw: # "rb" = bytes mode
    file_content = fraw.read()
chardet.detect(file_content)