How to write foreach %dopar% logs to separate files

Question

I use R to run Ant Colony Optimization and usually repeat the same optimization several times to cross-validate my results. I want to save time by running the processes in parallel with the foreach and doParallel packages.

A reproducible example of my code would be very long so I'm hoping this is sufficient. I think I managed to get the code running like this:

result <- list()

short <- function(n){
            for(n in 1:10){
               result[[n]] <- ACO(data, ...)}}

foreach(n=1:50) %dopar% short(n)

Within the ACO() function I continuously create objects with intermediate results (eg the current pheromone levels) which I save using write.table(..., append=TRUE) to keep track of the iterations and their results. Now that I'm running the processes in parallel, the file I write contains results from all processes and I'm not able to tell which process the data belongs to. Therefore, I'd like to write different files for each process.

What's the best way, in general, to save intermediate results when using parallel processing?

Answer 1

You can use the log4r package to write info needed in a log file. More info about the package here .

An example of the code which you have to put in your short function:

# Import the log4r package. 
library('log4r')
# Create a new logger object with create.logger(). 
logger <- create.logger()
# Set the logger's file output. 
logfile(logger) <- 'base.log'
# Set the current level of the logger. 
level(logger) <- 'INFO'
# Try logging messages with different priorities. # At priority level INFO, a call to debug() won't print anything. 
debug(logger, 'Iretation and result info')