How to extract random effects and variance components from lme4 wrapped in dlply

Question

This post How can I extract elements from lists of lists in R? answers some of my questions but that still doesn't quite work for me and what I need to do goes beyond my R knowledge.

I have data from field trials in 2 environments (=trials), 2 years and 5 traits of interest (defined by trait_id). GID is the unique line identifier. My model in lme4 is:

mods <- dlply(data,.(trial,trait_id), 
 function(d)
  lmer(phenotype_value ~(1|GID)+(1|year)+(1|year:GID)+(1|year:rep), 
       na.action = na.omit,data=d))

Running this returns a large list of 10 elements and I would like to store the random effects for GID for all traits per trial in a data frame. I tried several things:

blup=lapply(mods,ranef, drop = FALSE)
blup1=blup[[1]]
blup2=blup1$GID  

will give me a df with the random effects for one trait per trial, I was hoping for something more streamlined that will preserve some of info like $irrigation.GRYLD in the column names.

Here is a reproducible example with only two traits ( GRYLD , PTHT ), 2 years ( 11OBR , 12OBR ), and two reps:

structure(list(GID = structure(c(1L, 2L, 3L, 4L, 5L, 5L, 1L, 
2L, 4L, 3L, 1L, 2L, 3L, 4L, 5L, 5L, 1L, 2L, 4L, 3L, 1L, 2L, 3L, 
4L, 5L, 5L, 2L, 1L, 4L, 3L, 1L, 2L, 3L, 4L, 5L, 5L, 2L, 1L, 4L, 
3L, 1L, 2L, 3L, 4L, 5L, 5L, 1L, 2L, 4L, 3L, 1L, 2L, 3L, 4L, 5L, 
5L, 1L, 2L, 4L, 3L, 1L, 2L, 3L, 4L, 5L, 5L, 2L, 1L, 4L, 3L, 1L, 
2L, 3L, 4L, 5L, 5L, 2L, 1L, 4L, 3L), .Label = c("A", "B", "C", 
"D", "E"), class = "factor"), year = structure(c(1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("11OBR", 
"12OBR"), class = "factor"), trial = structure(c(1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("heat", 
"irrigation"), class = "factor"), rep = c(1L, 1L, 1L, 1L, 1L, 
2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 
2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 
1L, 1L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 
2L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L), trait_id = structure(c(1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("GRYLD", 
"PTHT"), class = "factor"), phenotype_value = c(3.93, 3.38, 1.65, 
4.33, 2.45, 2.48, 3.98, 3.3, 4.96, 1.53, 87.5, 69.5, 65.5, 84.5, 
77, 81, 94.5, 84.5, 89, 81, 6.56, 4.3, 5.76, 7.3, 5.73, 4.14, 
5.93, 6.96, 8.43, 5.81, 114.5, 100, 104.5, 110, 110, 106, 99, 
97.5, 105, 100, 0.119, 0.131, 0.681, 0.963, 0.738, 1.144, 0.194, 
0.731, 0.895, 0.648, 35, 50, 45, 50, 45, 50, 55, 45, 50, 55, 
2.79, 3.73, 3.96, 4.64, 5.03, 2.94, 3.78, 4.14, 3.89, 3.21, 90, 
95, 105, 100, 105, 85, 95, 100, 100, 95)), .Names = c("GID", 
"year", "trial", "rep", "trait_id", "phenotype_value"), class = "data.frame", row.names = c(NA, 
-80L))

Answer 1

I'm not quite sure what you want as an output format, but how about:

all_ranef <- function(object) {
    rr <- ranef(object)
    ldply(rr,function(x) data.frame(group=rownames(x),x,check.names=FALSE))
}
ldply(mods,all_ranef)

##         trial trait_id      .id   group   (Intercept)
## 1        heat    GRYLD year:GID 11OBR:A  7.935352e-01
## 2        heat    GRYLD year:GID 11OBR:B  1.960487e-01
## 3        heat    GRYLD year:GID 11OBR:C -1.504116e+00
## ...
## 82 irrigation     PTHT year:rep 12OBR:2 -1.595022e+00
## 83 irrigation     PTHT     year   11OBR  2.915033e+00
## 84 irrigation     PTHT     year   12OBR -2.915033e+00

this works reasonably well because all of your random effects are intercept-only. If you had some random-slopes terms in the models you might either want to reshape2:::melt() the individual random effects, or use rbind.fill() to combine data frames with different random-effects columns.

library("ggplot2"); theme_set(theme_bw())
ggplot(vals, aes(y=group,x=`(Intercept)`))+
    geom_point(aes(colour=interaction(trial,trait_id)))+
    facet_wrap(~.id,scale="free")

By the way, it's usually inadvisable to use a factor with only 2 levels ( YEAR ) as a grouping variable ...