无法从CSV文件读取非ASCII字符

Question

I am trying to read csv file which contains a sentence each line. Each sentence may contain foreign words such as Chinese characters. I want to remove or ignore those foreign characters and return only English words or ascii words.

Example of how the string may look like in the file:

'小心 Careful'

Desired output: Careful

import csv
from string import ascii_letters, punctuation

def remove_non_ascii(string):
    ascii_letters = set(ascii_letters)
    tokens = nltk.word_tokenize(string)
    ascii_words = [word for word in tokens if any(letter in ascii_letters for letter in word)]
    return ascii_words

with open(job_file, mode = 'r', encoding = 'utf8') as infile:   
    line_reader  = csv.reader(infile)
    for row in line_reader:
        new_line = remove_non_ascii(row[1])
        print (new_line)
        if row[1]:
            open(output_file, 'a', newline='', encoding = 'utf8') as outfile:
            line_writer = csv.writer(outfile)
            line_writer.writerow('')             

This is the error I get when I run that code.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 2848: invalid start byte

The error goes away if I change the encoding from utf8 to cp1252 because the Chinese characters are converted into '????'. Is it possible to remove those unwanted characters and return only ascii compliant characters?

Answer 1

If you are interested in the ascii part of your input file only, you can use

open(job_file, mode = 'r', encoding = 'ascii', errors = 'ignore')

This should ignore all characters that are not ascii compliant. The Python docs for open() give you more options you might want to look at.