从语料库中删除非ASCII

Question

I'm using NLTK for my project. However, if a non-ascii word like '•' exist. NLTK cannot tokenize it. I'm using nltk.word_tokenize as the tokenizer. How do I remove such words from entire corpus or make the tokenizer aware of such words?

Answer 1

Use the below code to remove nonascii from your corpus:

ip=open(nonascii.txt,'r')
#Edit should be in w mode
op=open(ascii.txt,'w')
for line in ip:
        line=line.strip().decode("ascii","ignore").encode("ascii")
        if line=="":continue
        op.write(line)
ip.close()
op.close()