R：在分组条件下聚合大数据帧

Question

I'm trying to figure out the fastest way to aggregate a large data frame (about 50M rows) that looks similar to:

>sample_frame = data.frame("id" = rep(sample(1:100,2,replace=F),3),
+ "date" = sample(seq(as.Date("2014-01-01"),as.Date("2014-02-13"),by=1),6),
+ "value" = runif(6))
> sample_frame
  id       date      value
1 73 2014-02-11 0.84197491
2  7 2014-01-14 0.08057893
3 73 2014-01-16 0.78521616
4  7 2014-01-24 0.61889286
5 73 2014-02-06 0.54792356
6  7 2014-01-06 0.66484848

Here we have 2 unique IDs with 3 dates and a value assigned to each. I know that I can use ddply, or data.table, or just a lapply to aggregate and find the mean for each ID.

What I'm really looking for is a way to quickly find the mean for each ID for the most recent two dates. For example, with sapply:

> sapply(split(sample_frame,sample_frame$id),function(x){
+   mean(x$value[x$date%in%x$date[order(x$date,decreasing=T)][1:2]])
+ })
        7        73 
0.3497359 0.6949492

I can't figure out how to get data.table to do this. Thoughts? Hints?

Answer 1

Why not use tail in your "data.table" aggregation step?

set.seed(1)
sample_frame = data.frame("id" = rep(sample(1:100,2,replace=F),3),
                          "date" = sample(seq(as.Date("2014-01-01"),
                                              as.Date("2014-02-13"),by=1),6),
                          "value" = runif(6))

DT <- data.table(sample_frame, key = "id,date")
DT
#    id       date      value
# 1: 27 2014-01-09 0.20597457
# 2: 27 2014-01-26 0.62911404
# 3: 27 2014-02-07 0.68702285
# 4: 37 2014-02-06 0.17655675
# 5: 37 2014-02-09 0.06178627
# 6: 37 2014-02-13 0.38410372
DT[, mean(tail(value, 2)), by = id]
#    id        V1
# 1: 27 0.6580684
# 2: 37 0.2229450

---

Since you require the mean of just two values, you can do it directly (without using mean ). And you can use the internal variable .N instead of tail to get more speed-up. You just have to take care of the case where there's just 1 date. Basically, this should be much faster.

DT[, (value[.N]+value[max(1L, .N-1)])/2, by=id]