Parallel computing in R with C-within-R functions, using snowfall package. Issue: Mac shows loading wheels and almost freezes

Question

I have a R package which contains C-within-R function, named myFun. I would like to call this myFun in my parallel computing. This myFun works fine by itself on my Mac, however when it is called via the parallel computing function sfClusterApplyLB, it shows strange behaviour: my Mac shows loading wheels and R almost freezes. After a while, R stops freezing and sfClusterApplyLB returns the parallelized results. I really want to avoid this freezing condition as I cannot even scroll up/down the R Console when it freezes!

To illustrate this, I have a small example codes.

I have a small C code, which loops 100 times while printing out the number of iterations every 20 sec and sleeping 1 sec at every iteration:

 # include <R.h>
 # include <Rinternals.h>
 # include <Rmath.h>
 # include <R_ext/Linpack.h>
 # include <unistd.h>

 SEXP myC (SEXP j)
  {
   for (int i = 0; i < 100; i++)
      {
        R_FlushConsole(); 
        R_ProcessEvents();
        R_CheckUserInterrupt(); 
        sleep(1); // sleep one second at each iteration. this sleep is
        // replaced by something in my code
        if (i%20==0) Rprintf("\v%d iterations are done...",i);
    }
  return (R_NilValue);
  }

I created a temporary R package "myRpack" which contains this myC function as well as its R wrapper function myFun, which is defined as:

 myFun <- function(x)
   {
    .Call("myC",
       as.integer(x),
       "myRpack")
    }

"myRpack" was installed to my Mac via the command R CMD INSTALL myRpack in terminal.

This function myCfun works fine when it is run independently. To see,

 library(myRpack)
 myFun(1)

Now, I want to parallel compute this myFun using snowfall package. Here is the R-code for parallel computing for this purpose:

 library("snowfall")
 sfInit(parallel=TRUE,cpus=3,type="SOCK")
 x <- 1 : 100
 res.list <- sfClusterApplyLB(x,myFun)

Then the R freezes!

PS I experienced this problem a while ago when I was executing C-within-R function (with NO parallel computing). I asked this question to Rcpp: Mac shows loading wheel and almost freeze . The solution that time was to add the lines

    R_FlushConsole(); 
    R_ProcessEvents();
    R_CheckUserInterrupt();  

in my C file (which I did in my code, too). However, this solution does not help in parallel computing environment...

I would appreciate any help.

PSS Even if I define the myC function as:

# include <R.h>
# include <Rinternals.h>
# include <Rmath.h>
# include <R_ext/Linpack.h>
# include <unistd.h>

SEXP myC (SEXP j)
{
  const int input=INTEGER(j)[0];
  Rprintf("\n input %d",input);
  for (int i = 0; i < 100; i++)
    {
  R_FlushConsole(); 
  R_ProcessEvents();
      R_CheckUserInterrupt(); 
  sleep(1); // sleep one second at each iteration. this sleep is
      // replaced by something in my code
  if (i%20==0) Rprintf("\v%d iterations are done...",i);
}
  return (R_NilValue);
}

The problem is present.

Answer 1

I encountered the same problem, and stumbled upon a workaround which may or may not work in your case.

tl;dr: I realized that scripts I was running, which called functions using lower-level C code in parallel, that would hang in the R GUI, would execute properly in an R instance in the Terminal (once time per instance—subsequent source() ings would hang).

---

More background, for anyone else facing the same issue: I ran into it calling the function gbm.step() the dismo package in parallel. I was using doParallel and foreach packages to parallelize it. The `gbm.step() function a C function to fit boosted regression tree models, and when I'd run it in parallel, it'd freeze (upon examination, the majority of CPU usage was System, not User).

I only started encountering this problem on Mavericks, so I wonder if it has to do with Mavericks' compressed memory or something similar.

(My ultimate workaround was to start running this code on a Linux server, so take that for what it's worth.)

Answer 2

In your myC function

SEXP myC (SEXP j) {

Shouldn't "j" be transformed somewhere into a C variable?