在R中存储和处理面板数据的有效方法

Question

Suppose there are time series data of 1024 individuals stored in separate csv files. I fread them into memory, obtaining 1024 data.frames , in following form

Tables$Individual1

SampleDate,var1,var2,var3,...
2001-01-01,1001,2001,3001,...
2001-01-02,1002,2002,3002,...
2001-01-03,1004,2004,3004,...
...
2017-01-01,9999,9999,9999,...

Tables$Individual2

SampleDate,var1,var2,var3,...
1992-03-01,1101,2101,3101,...
1992-03-02,1102,2102,3102,...
1992-03-03,1104,2104,3104,...
...
2017-01-01,8888,8888,8888,...

...

The tables have different initial observation dates because the individuals have different dates of birth but each subsequent day corresponds to a row in order. If I use an array to store the combined data, then many elements(days before birth) will be empty. What is the best way to organize them in memory that allows quick access to cross sectional data? For example, I want to fetch var1,var3 at 2010-04-01 of all individuals which exists at that day. Currently I have to sapply a function which extracts a part of each table and this is awfully slow.

Another matter. Lets say I need to sort these individuals by a function f(var1,var2,var3,...) at 8 different dates. Now this is an embarrassingly parallel task so I readily grab the parallel package, only to find that it takes forever to clusterExport those tables. Is there any clusterExport variant that utilizes shared memory, or maybe I should switch to linux to make fork clusters?

Any help will be appreciated.

Answer 1

Why not add a field with individualsID and put all the data into one dataframe.

Take df1 as your sample1 and df2 as your sample2, then

df1$IndID <- "01"

adds the Individual ID to the dataframe, which leads to

> df1
  SampleDate var1 var2 var3 IndID
1 2001-01-01 1001 2001 3001    01
2 2001-01-02 1002 2002 3002    01
3 2001-01-03 1004 2004 3004    01
4 2017-01-01 9999 9999 9999    01

Same with df2

df2$IndID <- "02"

And combine them to one dataframe

df <- rbind(df1,df2)

which leads to

> df
  SampleDate var1 var2 var3 IndID
1 2001-01-01 1001 2001 3001    01
2 2001-01-02 1002 2002 3002    01
3 2001-01-03 1004 2004 3004    01
4 2017-01-01 9999 9999 9999    01
5 1992-03-01 1101 2101 3101    02
6 1992-03-02 1102 2102 3102    02
7 1992-03-03 1104 2104 3104    02
8 2017-01-01 8888 8888 8888    02

then handling of the data is easy and timesaving - eg. your question - fetch var1 and var3

> df[df$SampleDate=="2017-01-01", c("var1","var3")]
  var1 var3
4 9999 9999
8 8888 8888

To sort the data ...

> library(dplyr)
> arrange(df, IndID, var1, var2, var3)

Answer 2

Partial answer:

lapply(Tables, '[[', 'var1')

Should return you a list containing the var1 column for each indvidual, you may be able to pass more arguments to the second portion to pull out only the required date values.