去除python中的無意義詞

Question

我想刪除數據集中的無意義詞。

我試過了，我看到 StackOverflow 是這樣的：

import nltk
words = set(nltk.corpus.words.words())

sent = "Io andiamo to the beach with my amico."
" ".join(w for w in nltk.wordpunct_tokenize(sent) \
     if w.lower() in words or not w.isalpha())

但是現在因為我有一個 dataframe 我如何在整個列上迭代它。

我試過這樣的事情：

import nltk
words = set(nltk.corpus.words.words())

sent = df['Chats']
df['Chats'] = df['Chats'].apply(lambda w:" ".join(w for w in 
nltk.wordpunct_tokenize(sent) \
     if w.lower() in words or not w.isalpha()))

但我收到錯誤 TypeError: expected string or bytes-like object

Answer 1

類似下面的內容將生成一個Clean列，它將您的 function 應用於Chats列

words = set(nltk.corpus.words.words())

def clean_sent(sent):
    return " ".join(w for w in nltk.wordpunct_tokenize(sent) \
     if w.lower() in words or not w.isalpha())

df['Clean'] = df['Chats'].apply(clean_sent)

要更新Chats列本身，您可以使用原始列覆蓋它：

df['Chats'] = df['Chats'].apply(clean_sent)

Answer 2

import re

df['Chats'] = [re.sub('\n', '', x) for x in df['Chats']]