error when fitting random effects model using bam() rather than gam() function in mgcv package, R

Question

I am fitting a model with many random effects using the bam() function within the mgcv package for R. My basic model structure looks like:

fit <- bam(y ~ s(x1) + s(x2) + s(xn) + s(plot, bs = 're'), data = dat)

This function works for 4 subsets of my data, but not the fifth, which is surprising. Instead, it throws this error:

Error in qr.qty(qrx, f) : 
  right-hand side should have 14195 not 14196 rows

This error goes away if I switch to using the gam() rather than bam() function. It also goes away if I drop the random effect from the model. I am really unsure whats causing this error, or what to do about it. Unfortunately, generating a reproducible example would require passing along a very large dataset, as its not clear why this error is thrown on this particular dataset, compared to 4 other datasets fitting the exact same model.

Any idea why this error is being thrown, and how to overcome it, would be greatly appreciated.

Answer 1

I had the same question and I found this r-help mail which tries to solve the same problem:

[R] bam (mgcv) not using the specified number of cores

After reading the mail, I deleted all the code about the cluster, such as the argument cluster in bam() function. Then the error message goes away.

I don't know the details but I hope this trick will help you.

Answer 2

One possible cause of

Error in qr.qty(qrx, f): right-hand side should have 14195 not 14196 rows

is running out of RAM. This may explain why you have seen the error for some datasets but not others. This is especially common when using a large cluster size.