Getting an error while executing perplexity function to evaluate the LDA model

Question

I am trying to evaluate the topic modeling(LDA). Getting a error while execting perplexity function as: Error in (function (classes, fdef, mtable) : unable to find an inherited method for function 'perplexity' for signature '"LDA_Gibbs", "numeric"' someone please help to solve this.

Answer 1

As you haven't provided any example of your code, it's difficult to know what your exact issue is. However, I found this question when I was facing the same error so I will provide the problem I faced and solution here in the hope that it may help someone else.

In the topicmodels package, when fitting using Gibbs the perplexity() function requires newdata to be supplied in a document-term format. If you give it something else, you get this error. Going by your error message you were probably giving it something numeric instead of a dtm.

Here is a working example, using the newsgroups data from the lda package converted to the dtm format:

library(topicmodels)

# load the required data from lda package
data("newsgroup.train.documents", "newsgroup.test.documents", "newsgroup.vocab", package="lda")


# create document-term matrix using newsgroups training data
dtm <- ldaformat2dtm(documents = newsgroup.train.documents, vocab = newsgroup.vocab)

# fit LDA model using Gibbs sampler
fit <- LDA(x = dtm, k = 20, method="Gibbs")

# create document-term matrix using newsgroups test data
testdtm <- ldaformat2dtm(documents = newsgroup.test.documents, vocab = newsgroup.vocab)

# calculate perplexity
perplexity(fit, newdata = testdtm)