How to remove stop-words list items from a text

Question

I try to remove stopwords from a dataframe with below code. It does not produce error but it does not remove stopwords from the dataframe.

def stop_words(df):

    stop_words = set(["a", "acaba", "altı","alti", "ama", "ancak","bir"])

    df['text'] = [word for word in df['text'] if word not in stop_words]
    return df.text

df.text = stop_words(df)

for instance df.text[2] is "gel sen necektigimi bir de bana sor".It does not remove "bir" word. How can I solve this ?

Answer 1

df['text'] is a column of strings. Iterate over it and you iterate over each sentence. What did you expect?

You'll need to split each word and iterate over that. You could use a list comprehension. You could also use apply :

def f(x):
    return [w for w in x if w not in stop_words]

df['text'] = df['text'].str.split().apply(f).str.join(' ')