我在我的 dataframe 上使用单词 tokenize function，通过编写 word_dict，但在执行错误消息“预期的字符串或类似字节的对象”之后

Question

I want to write the code word_dict , by calling the column name more_clean on the dataframe, but the error expected string or bytes-like object appears.

This is my dataframe:

And this is my code:

word_dict = {}
for i in range(0,len(df['more_clean'])):
    sentence = df['more_clean'][i]
    word_token = word_tokenize(sentence)
    for j in word_token:
        if j not in word_dict:
            word_dict[j] = 1
        else:
            word_dict[j] += 1

and an error message appears like this

TypeError: expected string or bytes-like object

Answer 1

You need to make sure the sentence variable is of a str type:

word_token = word_tokenize(str(sentence))

See the nltk.tokenize.word_tokenize documentation :

nltk.tokenize.word_tokenize(text, language='english', preserve_line=False)

Parameters

• text (str) – text to split into words
  
• language (str) – the model name in the Punkt corpus
  
• preserve_line (bool) – A flag to decide whether to sentence tokenize the text or not.