如何标记 R 中的文本列表

Question

I have a list of texts imported from 10 documents, such as:

library(quanteda)
library(readtext)
path <- "the working direction"
doc1 <- readtext(paste0(path, "/*_XXX.docx"))

view(doc1) looks like
[[1]] character(1) 'some words'
[[2]] character(2) 'some words'
...

Now, I need to tokenize this list of texts, so I used

tok_cov1 <- doc1 %>% 
  tokens(remove_punct = TRUE,
         remove_numbers = TRUE,
         remove_symbols = TRUE) %>% 
  tokens_tolower(keep_acronyms = TRUE) %>% 
  tokens_wordstem() %>% 
  tokens_remove(pattern = stopwords("en"))

The code did not return with any error, but would not tokenize anything. The doc1 still looks the same as untokenized.

I am aware that specifying 'doc1' as 'doc1[[n]]' would return with tokens from the corresponding texts, e,g.,

tok_cov1 <- doc1[[1]] %>% 
  tokens(remove_punct = TRUE,
         remove_numbers = TRUE,
         remove_symbols = TRUE) %>% 
  tokens_tolower(keep_acronyms = TRUE) %>% 
  tokens_wordstem() %>% 
  tokens_remove(pattern = stopwords("en"))

However, I'd need it works on every text instead of doing the text one by one. Any help is greatly appreciated. Thank you.

Answer 1

The quanteda corpus() function works directly on objects created by readtext() . So in your example, simply use:

library(quanteda)
corpus(doc1) %>%
  tokens()

adding your preferred options for the tokenisation of course.