Fixed effects regression in R (with a very large number of dummy variables)

Question

Is there an easy way to do a fixed-effects regression in R when the number of dummy variables leads to a model matrix that exceeds the R maximum vector length? Eg,

> m <- lm(log(bid) ~ after + I(after*score) + id, data = data)
Error in model.matrix.default(mt, mf, contrasts) : 
cannot allocate vector of length 905986769

where id is a factor (and is the variable causing the problem above).

I know that I could go through and de-mean all the data, but this throws the standard errors off (yes, you could compute the SE's "by hand" w/ a df adjustment but I'd like to minimize the probability that I'm introducing new errors). I've looked at the plm package but it seems only designed for classical panel data w/ a time component, which is not the structure of my data.

Answer 1

Plm will work fine for this sort of data. The time component is not required.

> library(plm)
> data("Produc", package="plm")
> zz <- plm(log(gsp)~log(pcap)+log(pc)+log(emp)+unemp, data=Produc, index=c("state"))
> zz2 <- lm(log(gsp)~log(pcap)+log(pc)+log(emp)+unemp+factor(state), data=Produc)
> summary(zz)$coefficients[,1:3]
              Estimate   Std. Error    t-value
log(pcap) -0.026149654 0.0290015755 -0.9016632
log(pc)    0.292006925 0.0251196728 11.6246309
log(emp)   0.768159473 0.0300917394 25.5272539
unemp     -0.005297741 0.0009887257 -5.3581508
> summary(zz2)$coefficients[1:5,1:3]
                Estimate   Std. Error    t value
(Intercept)  2.201617056 0.1760038727 12.5089126
log(pcap)   -0.026149654 0.0290015755 -0.9016632
log(pc)      0.292006925 0.0251196728 11.6246309
log(emp)     0.768159473 0.0300917394 25.5272539
unemp       -0.005297741 0.0009887257 -5.3581508

Answer 2

The fixest() package should help you here. You can for example efficiently within demean the factor:

library(fixest)
feols(log(bid) ~ after + I(after*score) | id, data = data)

With large datasets, this is much faster than plm(). To the best of my knowledge, the lfe package is not supported anymore? See the warning here: https://cran.r-project.org/web/packages/lfe/index.html