data.table object trying to effect ddply like operation using by()

Question

I've searched quite a bit for this to no avail, let alone trying to make it work myself, so without further a do:

I have this data.table :

 DT = data.table(date=rep(c(as.Date("2010-01-01"),as.Date("2010-01-02")), each=3),       bucket=rep(c("bucket1","bucket2","bucket3"),each=2),
kbucket=c("(0,.5]","(.5,1]","(1,1.5]","(1.5,2]","(1.5,2]","(2.5,3]"),vol=1:6,o=10:15,m=20:25)

which looks like:

         date  bucket kbucket vol  o  m
1: 2010-01-01 bucket1  (0,.5]   1 10 20
2: 2010-01-01 bucket1  (.5,1]   2 11 21
3: 2010-01-01 bucket2 (1,1.5]   3 12 22
4: 2010-01-02 bucket2 (1.5,2]   4 13 23
5: 2010-01-02 bucket3 (1.5,2]   5 14 24
6: 2010-01-02 bucket3 (2.5,3]   6 15 25

I've used ddply like so, on DF, the facsimile of DT, but which is a data frame:

out <- ddply(DF,.(date,bucket,kbucket),wrap_summarize)

where wrap_summarize is defined as:

 wrap_summarize = function(x)  
                  { 
                       out <-  summarize(  x,
                       N = length(x$date),
                       sumVol = sum(x$vol),
                       sumO = sum(x$o),
                       avgM = mean(x$m,na.rm=TRUE))

                  }

to get

        date  bucket kbucket N sumVol sumO avgM
1 2010-01-01 bucket1  (.5,1] 1      2   11   21
2 2010-01-01 bucket1  (0,.5] 1      1   10   20
3 2010-01-01 bucket2 (1,1.5] 1      3   12   22
4 2010-01-02 bucket2 (1.5,2] 1      4   13   23
5 2010-01-02 bucket3 (1.5,2] 1      5   14   24
6 2010-01-02 bucket3 (2.5,3] 1      6   15   25

This is the desired result.

The actual data has this structure but is hundreds of thousands of rows. Thus the need for data.table methods. So I try this:

test <- DT[,list(N=length(DT$date),sumVol=sum(DT$vol),sumO=sum(DT$o),avgM=mean(DT$m,na.rm=T)),
by=list(date,bucket,kbucket)]

only to get, which is clearly not what is desired:

         date  bucket kbucket N sumVol sumO avgM
1: 2010-01-01 bucket1  (0,.5] 6     21   75 22.5
2: 2010-01-01 bucket1  (.5,1] 6     21   75 22.5
3: 2010-01-01 bucket2 (1,1.5] 6     21   75 22.5
4: 2010-01-02 bucket2 (1.5,2] 6     21   75 22.5
5: 2010-01-02 bucket3 (1.5,2] 6     21   75 22.5
6: 2010-01-02 bucket3 (2.5,3] 6     21   75 22.5

I think i need to use .SD here, but at this point, i thought it best to ask and share this problem if not get the most efficient solution. Thanks in advance!

Answer 1

You're looking for this:

DT[,list(
    .N,
    sumVol=sum(vol),
    sumO=sum(o),
    avgM=mean(m,na.rm=T)
),by=list(date,bucket,kbucket)]

which gives

#          date  bucket kbucket N sumVol sumO avgM
# 1: 2010-01-01 bucket1  (0,.5] 1      1   10   20
# 2: 2010-01-01 bucket1  (.5,1] 1      2   11   21
# 3: 2010-01-01 bucket2 (1,1.5] 1      3   12   22
# 4: 2010-01-02 bucket2 (1.5,2] 1      4   13   23
# 5: 2010-01-02 bucket3 (1.5,2] 1      5   14   24
# 6: 2010-01-02 bucket3 (2.5,3] 1      6   15   25