如何使用 spacy 在 pandas 数据框中删除停用词并获取引理？

Question

I have a column of tokens in a pandas data frame in python. Something that looks like:

 word_tokens
 (the,cheeseburger,was,great)
 (i,never,did,like,the,pizza,too,much)
 (yellow,submarine,was,only,an,ok,song)

I want get two more new columns in this dataframe using the spacy library. One column that contains each row's tokens with the stopwords removed, and the other one containing the lemmas from the second column. How could I do that?

Answer 1

You're right about making your text a spaCy type - you want to transform every tuple of tokens into a spaCy Doc. From there, it is best to use the attributes of the tokens to answer the questions of "is the token a stop word" (use token.is_stop ), or "what is the lemma of this token" (use token.lemma_ ). My implementation is below, I altered your input data slightly to include some examples of plurals so you can see that the lemmatization works properly.

import spacy
import pandas as pd

nlp = spacy.load('en_core_web_sm')

texts = [('the','cheeseburger','was','great'),
         ('i','never','did','like','the','pizzas','too','much'), 
         ('yellowed','submarines','was','only','an','ok','song')]

df = pd.DataFrame({'word_tokens': texts})

The initial DataFrame looks like this:

	word_tokens
0	('the', 'cheeseburger', 'was', 'great')
1	('i', 'never', 'did', 'like', 'the', 'pizzas', 'too', 'much')
2	('yellowed', 'submarines', 'was', 'only', 'an', 'ok', 'song')

I define functions to perform the main tasks:

1. tuple of tokens -> spaCy Doc
2. spaCy Doc -> list of non-stop words
3. spaCy Doc -> list of non-stop, lemmatized words

def to_doc(words:tuple) -> spacy.tokens.Doc:
    # Create SpaCy documents by joining the words into a string
    return nlp(' '.join(words))

def remove_stops(doc) -> list:
    # Filter out stop words by using the `token.is_stop` attribute
    return [token.text for token in doc if not token.is_stop]

def lemmatize(doc) -> list:
    # Take the `token.lemma_` of each non-stop word
    return [token.lemma_ for token in doc if not token.is_stop]

Applying these looks like:

# create documents for all tuples of tokens
docs = list(map(to_doc, df.word_tokens))

# apply removing stop words to all
df['removed_stops'] = list(map(remove_stops, docs))

# apply lemmatization to all
df['lemmatized'] = list(map(lemmatize, docs))

The output you get should look like this:

	word_tokens	removed_stops	lemmatized
0	('the', 'cheeseburger', 'was', 'great')	['cheeseburger', 'great']	['cheeseburger', 'great']
1	('i', 'never', 'did', 'like', 'the', 'pizzas', 'too', 'much')	['like', 'pizzas']	['like', 'pizza']
2	('yellowed', 'submarines', 'was', 'only', 'an', 'ok', 'song')	['yellowed', 'submarines', 'ok', 'song']	['yellow', 'submarine', 'ok', 'song']

Based on your use case, you may want to explore other attributes of spaCy's document object ( https://spacy.io/api/doc ). Particularly, take a look at doc.noun_chunks and doc.ents if you're trying to extract more meaning out of text.

It is also worth noting that if you plan on using this with a very large number of texts, you should consider nlp.pipe : https://spacy.io/usage/processing-pipelines . It processes your documents in batches instead of one by one, and could make your implementation more efficient.

Answer 2

If you are working with spacy you should make your text a spacy type, so something like this:

 nlp = spacy.load("en_core_web_sm")
 text = topic_data['word_tokens'].values.tolist()
 text = '.'.join(map(str, text))
 text = nlp(text)

This makes it easier to work with. You can then tokenize the words like this

 token_list = []
    for token in text:
    token_list.append(token.text)

And Remove stop words like so.

token_list= [word for word in token_list if not word in nlp.Defaults.stop_words]

I haven't yet figured out the lemmatization part yet, but this is a start till then.