R：计算预定义词典中单词的频率

Question

I have a very large dataset that looks like this: one column contains names, the second column contains their respective (very long) texts. I also have a pre-defined dictionary that contains at least 20 terms. How can I count the number of times these key words occur in each row of my dataframe? I have tried str_detect,grep(l), and %>% like, and looped over each row, but the problem seems to be that I want to detect too many terms, and these functions stop working when I use 15+ terms or so.

Would be sooo happy if anyone could help me out with this!

col1<- c("Henrik", "Joseph", "Lucy")
col2 <- c("I am going to get groceries", "He called me at six.", "No, he did not")
df <- data.frame(col1, col2)```
dict <- c("groceries", "going", "me") #but my actual dictionary is much larger 

Answer 1

Create a unique identifier for your rows. Split your col2 by words, one in each row. Filter for only the select words in your dict. Then count by each row. Finally, combine with original df and set NA to Zeros for rows that don't have any words from your dict.

library(dplyr)

col1 <- c("A","B","A")
col2 <- c("I am going to get groceries", "He called me at six.", "No, he did not")
df <- data.frame(col1, col2, stringsAsFactors = FALSE)
dict <- c("groceries", "going", "me")

df <- df %>% mutate(row=row_number()) %>% select(row, everything())

counts <- df %>% tidyr::separate_rows(col2) %>% filter(col2 %in% dict) %>% group_by(row) %>% count(name = "counts")

final <- left_join(df, counts, by="row") %>% tidyr::replace_na(list(counts=0L))
final
#>   row col1                        col2 counts
#> 1   1    A I am going to get groceries      2
#> 2   2    B        He called me at six.      1
#> 3   3    A              No, he did not      0

Answer 2

Here is a base R option using gregexpr

dfout <- within(
  df,
  counts <- sapply(
    gregexpr(paste0(dict, collapse = "|"), col2),
    function(x) sum(x > 0)
  )
)

or

dfout <- within(
  df,
  counts <- sapply(
    regmatches(col2, gregexpr("\\w+", col2)),
    function(v) sum(v %in% dict)
  )
)

which gives

> dfout
  col1                        col2 counts
1    1 I am going to get groceries      2
2    2        He called me at six.      1
3    3              No, he did not      0

Data

structure(list(col1 = 1:3, col2 = c("I am going to get groceries", 
"He called me at six.", "No, he did not")), class = "data.frame", row.names = c(NA, 
-3L))

Answer 3

I think my solution gives you the output you want - that is for each word in your "dict" list, you can see how many times it appears in each sentence. Each row is an entry in df$col2 ie a sentence. "Dict" is your vector of terms that you're looking to match. We can loop over the vector and for each entry in the vector we match how many times that entry appears in each row/sentence using stringr::str_count. Note the syntax for str_count: str_count(string being checked over, expression you're trying to match)

str_count returns a vector showing how many times the word appears in each row. I create a data frame of these vectors which will contain the same number of rows as there are entries in the dict vector. Then you can just cbind "dict" to that data frame and you can see how many times each word is used in each sentence. I adjust the column names at very end so you can match the words to the sentence #'s. Note that if you want to calculate row means you'll need to subset out the "dict" column of the final data frame because it's character.

 library(stringr)
 col1<- c("Henrik", "Joseph", "Lucy")
 col2 <- c("I am going to get groceries", "He called me at six.", "No, he    
 did not")
 df <- data.frame(col1, col2)
 dict <- c("groceries", "going", "me")

 word_matches <- data.frame()
 for (i in dict) {
 word_tot<-(str_count(df$col2, i))
 word_matches <- rbind(word_matches,word_tot)
 }
 word_matches
 colnames(word_matches) <- paste("Sentence", 1:ncol(word_matches))
 cbind(dict,word_matches)


        dict Sentence 1    Sentence 2    Sentence 3
 1 groceries        1           0           0
 2     going        1           0           0
 3        me        0           1           0