I am looking for a solution to remove the English stop words using NLTK corpus on a Pandas dataframe text column. Can we do it with the dataframe apply method, if yes, then please share it?
stop_words = set(stopwords.words('english'))
data['text'] = data['text'].apply(lambda text: " ".join(w) for w in text.lower().split() if w not in stop_words)
Thanks and appreciate it if someone can answer it.
You could tokenize your text column (or simply split into a list of words) and then remove the stop words using the map
or apply
method.
For example:
data = pd.DataFrame({'text': ['a sentence can have stop words', 'stop words are common words like if, I, you, a, etc...']})
data
text
0 a sentence can have stop words
1 stop words are common words like if, I, you, a...
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer('\w+')
stop_words = stopwords.words('english')
def clean(x):
doc = tokenizer.tokenize(x.lower())
return [w for w in doc if w in stop_words]
data.text.map(clean)
0 [sentence, stop, words]
1 [stop, words, common, words, like, etc]
Name: text, dtype: object
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.