Panel Data Regression on Biglm in R - Non Linear Model

Question

I have following data set which is panel data. Total data is about 78 millions rowcount. I have few more columns of data which I have skipped here.

                 date      stockName   PRC VOLUME
2 2016-06-01 09:30:53 ABCD IS Equity 14.25  13957
3 2016-06-01 09:30:54 EFGH IS Equity 14.25  14620
4 2016-06-01 09:31:04 IJKL IS Equity 14.25  14120
5 2016-06-01 09:31:11 MNOP IS Equity 14.25  13820
6 2016-06-01 09:31:47 ABCD IS Equity 14.30  20408
7 2016-06-01 09:31:58 EFGH IS Equity 14.30  29776

As far I understood plain biglm run is not for panel data. Please correct me if I am wrong. So how can I use it for panel data. Any kind of comments or suggestions are welcome.

Answer 1

If what you want to do is estimate a linear model with fixed effects, one possibility is to do it in two steps.

• Step 1: Within-transformation of your outcome and covariates: build within-transformed variables X and Y by subtracting the individual average of each variable from itself

• Step 2: Regression of the within-transformed variables using biglm

Answer 2

The page Econometrics at CRAN can give you an overview about the packages avaiable for econometric analysis.

As suggestion, I think lme4 , nlme and even pglm may be the packages for what you are looking for: nonlinear panel data, despite I don't know much about their performance when you have too much rows.

Although they are written in the mixed-effect models jargon, the plm vignette gives brief comments about the interchangeability between this terminology and that used by econometricians.