R: having trouble using quanteda corpus with readtext

Question

After reading my corpus with the Quanteda package, I get the same error when using various subsequent statements:

Error in UseMethod("texts") : no applicable method for 'texts' applied to an object of class "c('corpus_frame', 'data.frame')").

For example, when using this simple statement: texts(mycorpus)[2] My actual goal is to create a dfm (which give me the same error message as above).

I read the corpus with this code:

`mycorpus < corpus_frame(readtext("C:/Users/renswilderom/Documents/Stuff Im 
working on at the moment/Newspaper articles DJ/test data/*.txt", 
docvarsfrom="filenames", dvsep="_", docvarnames=c("Date of Publication", 
"Length LexisNexis"), encoding = "UTF-8-BOM"))`

My dataset consists of 50 newspaper articles, including some metadata such as the date of publication.

See screenshot.

Why am I getting this error every time? Thanks very much in advance for your help!

Response 1:

When using just readtext() I get one step further and texts(text.corpus)[1] does not yield an error.

However, when tokenizing, the same error occurs again, so:

token <- tokenize(text.corpus, removePunct=TRUE, removeNumbers=TRUE, ngrams 
= 1:2)
tokens(text.corpus)

Yields:

Error in UseMethod("tokenize") : no applicable method for 'tokenize' applied to an object of class "c('readtext', 'data.frame')"

Error in UseMethod("tokens") : no applicable method for 'tokens' applied to an object of class "c('readtext', 'data.frame')"

Response 2:

Now I get these two error messages in return, which I initially also got, so I started using corpus_frame()

Error in UseMethod("tokens") : no applicable method for 'tokens' applied to an object of class "c('corpus_frame', 'data.frame')"

In addition: Warning message: 'corpus' is deprecated. Use 'corpus_frame' instead. See help("Deprecated")

Do I need to specify that 'tokenization' or any other step is only applied to the 'text' column and not to the entire dataset?

Response 3:

Thank you, Patrick, this does clarify and brought me somewhat further. When running this:

# Quanteda - corpus way
readtext("C:/Users/renswilderom/Documents/Stuff Im working on at the moment/Newspaper articles DJ/test data/*.txt",
         docvarsfrom = "filenames", dvsep = "_", 
         docvarnames = c("Date of Publication", "Length LexisNexis", "source"), 
         encoding = "UTF-8-BOM")  %>%
  corpus() %>%
  tokens(removePunct = TRUE, removeNumbers = TRUE, ngrams = 1:2)

I get this:

Error in tokens_internal(texts(x), ...) : the ... list does not contain 3 elements In addition: Warning message: removePunctremoveNumbers is deprecated; use remove_punctremove_numbers instead

So I changed it accordingly (using remove_punct and remove_numbers ) and now the code runs well.

Alternatively, I also tried this:

# Corpus - term_matrix way
readtext("C:/Users/renswilderom/Documents/Stuff Im working on at the moment/Newspaper articles DJ/test data/*.txt",
         docvarsfrom = "filenames", dvsep = "_", 
         docvarnames = c("Date of Publication", "Length LexisNexis", "source"), 
         encoding = "UTF-8-BOM")  %>%
  term_matrix(drop_punct = TRUE, drop_numbers = TRUE, ngrams = 1:2)

Which gives this error:

Error in term_matrix(., drop_punct = TRUE, drop_numbers = TRUE, ngrams = 1:2) : unrecognized text filter property: 'drop_numbers'

After removing drop_numbers = TRUE , the matrix is actually produced. Thanks very much for your help!

Answer 1

To clarify the situation:

Versions 0.9.1 of the corpus package had a function called corpus . quanteda also has a function called corpus . To avoid the name clash between the two packages, the corpus corpus function got deprecated and renamed to corpus_frame in version 0.9.2; it was removed in version 0.9.3.

To avoid the name clash with quanteda , either upgrade to corpus to the latest version on CRAN (0.9.3), or else do

library(corpus)
library(quanteda)

Instead of the other order.

---

Now, if you want to use quanteda to tokenize your texts, follow the advice given in Ken's answer:

readtext("C:/Users/renswilderom/Documents/Stuff Im working on at the moment/Newspaper articles DJ/test data/*.txt",
     docvarsfrom = "filenames", dvsep = "_", 
     docvarnames = c("Date of Publication", "Length LexisNexis"), 
     encoding = "UTF-8-BOM"))  %>%
    corpus() %>%
    tokens(remove_punct = TRUE, remove_numbers = TRUE, ngrams = 1:2)

You may want to use the dfm function instead of the tokens function if your goal is to get a document-by-term count matrix.

If you want to use the corpus package, instead do

readtext("C:/Users/renswilderom/Documents/Stuff Im working on at the moment/Newspaper articles DJ/test data/*.txt",
     docvarsfrom = "filenames", dvsep = "_", 
     docvarnames = c("Date of Publication", "Length LexisNexis"), 
     encoding = "UTF-8-BOM"))  %>%
    term_matrix(drop_punct = TRUE, drop_number = TRUE, ngrams = 1:2)

Depending on what you're trying to do, you might want to use the term_stats function instead of the term_matrix function.

Answer 2

OK, you are getting this error because (as the error message states) there is no tokens() method for a readtext object class, which is a special version of a data.frame. (Note: tokenize() is older, deprecated syntax that will be removed in the next version - use tokens() instead.)

You want this:

library("quanteda")
library("readtext")
readtext("C:/Users/renswilderom/Documents/Stuff Im working on at the moment/Newspaper articles DJ/test data/*.txt",
         docvarsfrom = "filenames", dvsep = "_", 
         docvarnames = c("Date of Publication", "Length LexisNexis"), 
         encoding = "UTF-8-BOM"))  %>%
    corpus() %>%
    tokens(removePunct = TRUE, removeNumbers = TRUE, ngrams = 1:2)

It's the corpus() step you omitted. corpus_frame() is from a different package (my friend Patrick Perry's corpus ).