计算文本中停用词的数量

Question

I was wondering if anyone could help me with the following problem: I am trying to determine the number (count) of stop words in customer review texts. I am using the "quanteda" package stop words list in R. I have tokenised the text and filtered out the stop words by using the following code:

stop.words <- tokens_select(corpus2.tokens, stopwords())

However, I am now having trouble saving these results in such a way that I can count the actual number of stopwords included in each review.

Any tipps would be greatly appreciated. Thanks in advance!

Answer 1

You can use str_detect from stringr (or stri_detect from stringi ) to count the number of stopwords. str_detect will return TRUE or FALSE and these you can just count. Depending on which stopword list you have you can get different results. stopwords("en") from stopwords package will return 28. If you use stopwords(source = "smart") you will get a count of 61.

text <- "I've never had a better pulled pork pizza! The amount of toppings that they layered on it was astounding...bacon, corn, more pulled pork, and the sauce was delicious. I shared my pizza with 2 other people. I can't wait to go back."
stopwords <- stopwords::stopwords("en")

sum(stringr::str_detect(tolower(text), stopwords))
28