调用nlp时发生Python Spacy错误：UnicodeDecodeError：'ascii'编解码器无法解码字节0xe2

Question

Python3.6: I am using Spacy on a column of text in a pandas df. The text does have "Special Characters" and I need to keep them. nlp required unicode for some reason. I am getting an error from nlp below:

Any help would be very much appreciated.

# -*- coding: utf-8 -*-
import spacy
nlp = spacy.load("en_core_web_sm")

df['TextCol'] = df['TextCol'].str.encode('utf-8')
def function(row):
    doc = nlp(unicode(text))

df.apply(function, axis=1)

Return from nlp:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 

Answer 1

So I solved my own question. Not really sure what changed I switched IDEs from pycharm to eclipse(pydev). I am still using the same interpreter. Here is the changes, looks pretty standard usage.

# -*- coding: utf-8 -*-
import spacy
nlp = spacy.load("en_core_web_sm")

# Removed encode
# df['TextCol'] = df['TextCol'].str.encode('utf-8')
def function(row):
    # Removed unicode
    doc = nlp(text)

df.apply(function, axis=1)