ff: returning multiple arrays with a single ffapply function call

Question

I am dealing with a large dataset of 3D imaging data that I have loaded in to R using ff() .

require(ff)

nSubj <- 125
vol_dim <- c(139,137,87)
ff_qmap <- ff(0, dim=c(vol_dim,nSubj)

Simple calls like getting an average array/"volume" back work fine:

mean_qmap_vol <- ffapply(X=ff_qmap,MARGIN=c(1,2,3),AFUN=mean,RETURN=TRUE)

However, in some instances I would like to return more than one array/"volume" back in a single ffapply call; for instance, when performing some basic regression eg against age:

pval_vol <- ffapply( AFUN=f <- function(x) {
                  df$voxel <- x
                    fe1 <- lm(formula = voxel ~ age, df)
                    summary_fe1 <- summary(fe1)
                fe1_estimate <- summary_fe1$coefficients[2,1]
                fe1_pval <- summary_fe1$coefficients[2,4]
                return(fe1_pval)
}, X = ff_qmap, MARGIN = c(1,2,3), RETURN = TRUE)

This works for returning a single volume back, ie fe1_pval .

Is there a way to return both the fe1_estimate and fe1_pval (and perhaps more estimates) in one ffapply call?

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.2 LTS   
...
other attached packages:
[1] ff_2.2-13        bit_1.1-12       lme4_1.1-17      Matrix_1.2-8     ggplot2_2.2.1    fslr_2.12        neurobase_1.13.2
[8] oro.nifti_0.9.1 

Answer 1

I tried a number of solutions including returning a combined vector with c() and alternatively a list. However, I could not find a solution involving the ffapply routine that would work. Some key references I looked at are here:

• http://r.789695.n4.nabble.com/Looping-an-lapply-linear-regression-function-td4675475.html
• http://r.789695.n4.nabble.com/How-to-return-multiple-values-in-a-function-td858528.html
• extracting linear model coefficients into a vector within a loop

I found a stop-gap solution taking a classic for loop approach and cycling through the 3D dataset. Because the size of my array is not prohibitively large in this case, it works. I would ultimately prefer a solution using ffapply() so that it is extensible for higher resolution and larger datasets; and has the potential for parallelization. Open to suggestions!

The coef() stats function turned out to be a great way of extracting all the model coefficients in a standard way.

testlist <- vector(mode="list", length=vol_dim[1]*vol_dim[2]*vol_dim[3])
i <- 1
for (x in 1:vol_dim[1]) {
  for (y in 1:vol_dim[2]) {
    for (z in 1:vol_dim[3]) {
      df$voxel <- ff_logjac[x,y,z,]
      fe1 <- lm(formula = voxel ~ age, df)
      testlist[[i]] <- coef(summary(fe1))
      i <- i + 1
    }
  }
}

This is how the list() of lm coefficients is accessed after:

> length(testlist)
[1] 1656741
> vol_dim[1]*vol_dim[2]*vol_dim[3]
[1] 1656741
> testlist[[1]]
                Estimate  Std. Error    t value  Pr(>|t|)
(Intercept)  0.061286603 0.168853045  0.3629582 0.7191810
age         -0.002272307 0.003510186 -0.6473466 0.5223308
> testlist[[1656741]]
                Estimate  Std. Error   t value   Pr(>|t|)
(Intercept) -0.444810783 0.192135240 -2.315092 0.02763245
age          0.007246639 0.003994186  1.814297 0.07964480
> testlist[[1]][1,1]
[1] 0.0612866