读取CSV文件并验证基于UTF-8字符的列

Question

I have to read a CSV file which is containing columns (PersonName, age, address) and I have to validate the the PersonName. "PersonName may only contain UTF-8 characters."

I am using python3.x so cant use decode method after opening the file.

Please tell me how to open and read the file so that PersonName who is not containing any UTF-8 character can be ignored and I can move to next line for validation.

Answer 1

Assuming the rest of the file requires no checking or is UTF-8 legal (which includes ASCII data), you can open the file with encoding='utf-8' and errors='replace' . This will change any invalid bytes (in UTF-8 encoding) into the Unicode replacement character, \� . Alternatively, to preserve the data, you can use 'surrogateescape' as the errors handler, which uses private use Unicode codes to represent the original value in a way that can be undone later. You can then check for those as you go:

with open(csvname, encoding='utf-8', errors='replace', newline='') as f:
    for PersonName, age, address in csv.reader(f):
        if '\ufffd' in PersonName:
            continue
        ... PersonName was decoded without errors, so process the row ...

Or with surrogateescape , you can ensure any non-UTF-8 data (if that's "possible") in the other fields is restored on write:

with open(incsvname, encoding='utf-8', errors='surrogateescape', newline='') as inf,\
     open(outcsvname, 'w', encoding='utf-8', errors='surrogateescape', newline='') as outf:
    csvout = csv.writer(outf)
    for PersonName, age, address in csv.reader(f):
        try:
            # Check for surrogate escapes, and reject PersonNames containing them
            # Most efficient way to do so is a test encode; surrogates will fail
            # to encode with default error handler
            PersonName.encode('utf-8')
        except UnicodeEncodeError:
            continue  # Had non-UTF-8, skip this row

        ... PersonName was decoded without surrogate escapes, so process the row ...

        # You can recover the original file bytes in your code for a field with:
        #     fieldname.encode('utf-8', errors='surrogateescape')
        # Or if you're just passing data to a new file, write the same strings
        # back to a file opened with the same encoding/errors handling; the surrogates
        # will be restored to their original values:
        csvout.writerow([PersonName, age, address])