在R中运行LDA主题模型时如何使用8个内核

Question

I am running a Latent Dirichlet topic model in R using the follwing code:

for(k in 2:30) {
    ldaOut <-LDA(dtm,k, method="Gibbs", 
                 control=list(nstart=nstart, seed = seed, best=best, 
                              burnin = burnin, iter = iter, thin=thin))
    assign(paste("ldaOut", k, sep = "_"), ldaOut)
}

The dtm has 12 million elements, and each loop takes up to two hours on average. Meanwhile, R uses only 1 of my 8 logical processors ( i have i7-2700K CPU @ 3.50GHz wtih 4 cores). How can I make R use all the computational power available when I run one LDA topic model or when using a loop (as in this code)?

Thank you

EDIT: follwing gc_'s advice, I used the following code:

library(doParallel)

    n.cores <- detectCores(all.tests = T, logical = T) 
    cl <- makePSOCKcluster(n.cores) 

doParallel::registerDoParallel(cl)

burnin <- 4000 
iter <- 2000
thin <- 500 
seed <-list(2003,10,100,10005,765)
nstart <- 5 
best <- TRUE 

var.shared <- c("ldaOut", "dtm", "nstart", "seed", "best", "burnin", "iter", "thin", "n.cores")
library.shared <- "topicmodels" # Same for library or functions.


ldaOut <- c()

    foreach (k = 2:(30 / n.cores - 1), .export = var.shared, .packages = library.shared) %dopar% {
        ret <- LDA(dtm, k*n.cores , method="Gibbs", 
                   control=list(nstart=nstart, seed = seed, best=best, 
                                burnin = burnin, iter = iter, thin=thin))
        assign(paste("ldaOut", k*n.cores, sep = "_"), ret)
    }

The code ran without errors, but now there are 16 "R for Windows front-end" processes, 15 of which use 0% of the CPU and 1 is using 16-17%...And when the process was over i got this message:

A LDA_Gibbs topic model with 16 topics.

    Warning messages:
    1: In e$fun(obj, substitute(ex), parent.frame(), e$data) :
      already exporting variable(s): dtm, nstart, seed, best, burnin, iter, thin, n.cores
    2: closing unused connection 10 (<-MyPC:11888) 
    3: closing unused connection 9 (<-MyPC:11888) 
    4: closing unused connection 8 (<-MyPC:11888) 
    5: closing unused connection 7 (<-MyPC:11888) 
    6: closing unused connection 6 (<-MyPC:11888) 
    7: closing unused connection 5 (<-MyPC:11888) 
    8: closing unused connection 4 (<-MyPC:11888) 
    9: closing unused connection 3 (<-MyPC:11888) 

Answer 1

You can use the library doParallel

library(doParallel)

To get the number of cores of your computer:

n.cores <- detectCores(all.tests = T, logical = T) 

You can see the distinction between logical and physical cores.

Now you need to assign the core and set up all the process:

cl <- makePSOCKcluster(n.cores) 
doParallel::registerDoParallel(cl)

You can create more processes than you have cores on your computer. As R is creating new processes you need to define the library and variables you need to share with the workers.

var.shared <- c("ldaOut", "dtm", "nstart", "seed", "best", "burnin", "iter", "thin", "n.cores")
library.shared <- c() # Same for library or functions.

Then the loop will change to:

 ldaOut <- #Init the output#

 foreach (k = 2:(30 / n.cores - 1), .export = var.shared, .packages = library.shared)) %dopar% {
      ret <- LDA(dtm, k*n.cores , method="Gibbs", 
                     control=list(nstart=nstart, seed = seed, best=best, 
                                  burnin = burnin, iter = iter, thin=thin))
      assign(paste("ldaOut", k*n.cores, sep = "_"), ret)
}

I have never used LDA before so you might need to modify a bit the code above in order to make it works.

Answer 2

I think lda is hard to do in parallel, since each sweep uses the result of the previous sweep.

So to speed things up, you could imo

- reduce your dtm
- use faster libraries e.g. vowpal wabbit 
- use faster hardware e.g. aws

If you optimize for "hyperparameters" like alpha, eta, burnin, etc., you could run the full lda with different hyperparams on each core.