完全比较R？中具有相同目标的不同方法的性能

Question

In R , how can I cleanly compare different solutions to a same problem, being "fair" between each of them? Could running a resource-consuming solution before others alter the performences of the latter ones? How could one 'clean' the state of the machine between each test?

---

Suppose I want to compute the mean over columns of a matrix, I could do it the easy or the complicated way:

set.seed(9)
N = 1e7
ncol = 1e3
myT = matrix(runif(N), ncol = ncol)

func1 <- function(mat) {
  colMeans(mat)
}

func2 <- function(mat) {
  apply(mat, 2, function(x) sum(x)/length(x))
}

func3 <- function(mat) {
  nrows = c()
  for (i in 1:nrow(mat)) {
    nrows = c(nrows, 1) # yes, this is very stupid ;-)
  }
  colSums(mat) / sum(nrows)
}


system.time( replicate(1, t1 <- func1(myT)))
# user  system elapsed 
# 0.012   0.000   0.011 
system.time( replicate(1, t2 <- func2(myT)))
# user  system elapsed 
# 0.136   0.036   0.170
system.time( replicate(1, t3 <- func3(myT)))
# user  system elapsed 
# 0.140   0.032   0.170

Running several times the system.time() execution can give different results for a same test (possibly altering the conclusions). I noticed it was especially the case for more complicated, resource-sonsuming solutions, while the cleanest ones tend to have a more consistent execution time - what is the reason for this? How to avoid big changes between executions of the same expression, and how to prevent them to interfere with each other?

Is a call to gc() between tests useful, and is it enough?

I also know about the microbenchmark package, but I am looking for something more 'manual' in order to understand what happens.

I am working with RStudio , in case it matters...

Answer 1

The microbenchmark was design for this. system.time() is not as detailed

set.seed(9)
N = 1e5
ncol = 1e3
myT = matrix(runif(N), ncol = ncol)

library(microbenchmark)
microbenchmark(
  colmeans = colMeans(myT),
  wrong_apply = apply(myT, 2, function(x) sum(x)/length(x)), # wrong in case of NA
  correct_apply = apply(myT, 2, mean, na.rm = TRUE), # correct in case of NA
  stupid = {
    nrows = c()
    for (i in 1:nrow(myT)) {
      nrows = c(nrows, 1) # yes, this is very stupid ;-)
    }
    colSums(myT) / sum(nrows)
  }
)

Output

Unit: microseconds
          expr      min       lq       mean   median       uq       max neval cld
      colmeans   87.235   92.367   96.44175   95.787   98.781   129.142   100 a  
   wrong_apply 3004.886 3071.595 3483.02090 3166.739 3267.445 18707.947   100  b 
 correct_apply 7595.387 7895.148 8850.87886 8106.179 8461.745 13928.438   100   c
        stupid  144.109  156.510  166.15237  163.351  171.690   255.290   100 a