从Python的字符串列中删除停用词

Question

I'm working on a project to read the text and make a prediction of the outcome. As part of cleaning the data I am trying to remove all of the stopwords. When I try to do this, I need the output to be in a datafram format but I am running into issues there.

So, after much cleaning I got the data to the point where it looks like this.

The labels are in a different dataframe that I would have to merge but that is besides the point.

What I am trying to do now is remove all of the stopwords from each string in each row.

After some research the code I am using looks like this:

import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
stop_words = set(stopwords.words('english'))
ht_comments_only_no_stop['All_Comments'] = ht_comments_only_summary['All_Comments'].apply(lambda x: [item for item in x if item not in stop_words])

The ht_comments_only_summary is basically what you see in the first picture above.

The problem is that now when I try looking at "ht_comments_only_no_stop" I see:

But what I need is the output to just look like the first picture in dataframe format minus all the stopwords under the "All_Comments" column.

Any help would be greatly appreciated.

Answer 1

Ok, I figured it out.

First, there is a different issue which was that I needed to break down the strings into a list of words.

Then after that I can successfully remove the stopwords.

Finally I was able to convert the output back into a dataframe.

Best