带有 lme4 或其他包的稀疏混合模型

Question

I use mixed models on a large file (500000 rows). My model formula looks like this:
Y ~ 0 + num1:factor1 + num1:factor2 + num2:factor3 + factor4 + (0 + num3|subject) + (0 + num4|subject) + (1|subject) ,
where num - numeric variables; factor - categorical variables/factors.

Since categorical variables have many unique levels , the fixed effects matrix is ​​very sparse (sparsity ~0.9).
Fitting such a matrix if it is handle as dense requires a lot of time and RAM.

I had the same problem with linear regression.
My dense matrix was 20GB , but when I converted it to sparse it became only 35 MB .
So, I refused to use lm function and instead it used two another functions:

1. sparse.model.matrix (to create a sparse model/design matrix) and
2. MatrixModels:::lm.fit.sparse (to fit a sparse matrix and calculate coefficients).

Can I apply a similar approach to mixed models?
What functions / packages can I use to implement this?

That is, my main question is whether it is possible to implement mixed models with sparse matrices ?
What functions should I use to create X and Z sparse model matrices?
Then, which function should I use for fitting the model with sparse matrices to get coefficients?

I would be very-very grateful for any help with this!

Answer 1

• As of version 1.0.2.1 on CRAN, glmmTMB has a sparseX argument:

sparseX: a named logical vector containing (possibly) elements named "cond", "zi", "disp" to indicate whether fixed-effect model matrices for particular model components should be generated as sparse matrices, eg 'c(cond=TRUE)'. Default is all 'FALSE'

You would probably want glmmTMB([formula], [data], sparseX=c(cond=TRUE)) ( glmmTMB uses family="gaussian" by default).

glmmTMB is not quite as fast for linear mixed models as lme4 is: I don't know what your mileage will be (but will be interested to here). There is also some discussion here about how to hack the equivalent of sparse model matrices in lme4 (by letting the many-level factor be a random effect with a large fixed variance).