R data.table group由多列组成1列和求和

Question

I have the following data.table :

> dt = data.table(sales_ccy = c("USD", "EUR", "GBP", "USD"), sales_amt = c(500,600,700,800), cost_ccy = c("GBP","USD","GBP","USD"), cost_amt = c(-100,-200,-300,-400))
> dt
   sales_ccy sales_amt cost_ccy cost_amt
1:       USD       500      GBP     -100
2:       EUR       600      USD     -200
3:       GBP       700      GBP     -300
4:       USD       800      USD     -400

My aim is to get the following data.table :

> dt
   ccy total_amt
1: EUR       600
2: GBP       300
3: USD       700

Basically, I want to sum all costs and sales together by currency. In reality, this data.table has >500,000 rows so I would want a fast and efficient way to sum the amounts together.

Any idea of a fast way to do this?

Answer 1

Using data.table v1.9.6+ which has improved version of melt which can melt in to multiple columns simultaneously,

require(data.table) # v1.9.6+
melt(dt, measure = patterns("_ccy$", "_amt$")
    )[, .(tot_amt = sum(value2)), keyby = .(ccy=value1)]

Answer 2

You can consider merged.stack from my "splitstackshape" package.

Here, I've also used "dplyr" for piping, but you can skip that if you prefer.

library(dplyr)
library(splitstackshape)

dt %>%
  mutate(id = 1:nrow(dt)) %>%
  merged.stack(var.stub = c("ccy", "amt"), sep = "var.stubs", atStart = FALSE) %>%
  .[, .(total_amt = sum(amt)), by = ccy]
#    ccy total_amt
# 1: GBP       300
# 2: USD       700
# 3: EUR       600

---

The development version of "data.table" should be able to handle melting groups of columns. It's also faster than merged.stack .

Answer 3

Even dirtier than @Pgibas 's solution:

dt[,
   list(c(sales_ccy, cost_ccy),c(sum(sales_amt), sum(cost_amt))), # this will create two new columns with ccy and amt
   by=list(sales_ccy, cost_ccy)  # nro of rows reduced to only unique combination ales_ccy, cost_ccy
  ][,
    sum(V2), # this will aggregate the new columns
    by=V1
    ]

Benchmark

I did a couple of test to check my code against the solution with Data Table 1.9.5 suggested by Arun.

Just an observation, I just generated 500K+ rows duplicating the original data.table, this reduced the number of couple sales_ccy/cost_ccy, which reduced also the number of row crunched by the second data.table [] (just 8 rows created in this scenario).

I don't think that in a real world scenario the number of rows returned will be near 500K+ (probably, but I studied these thing a while ago, N^2 where N is the number of currency used), but it's still something to keep in mind looking at these results.

library(data.table)
library(microbenchmark)

rm(dt)
dt <- data.table(sales_ccy = c("USD", "EUR", "GBP", "USD"), sales_amt = c(500,600,700,800), cost_ccy = c("GBP","USD","GBP","USD"), cost_amt = c(-100,-200,-300,-400))
dt


for (i in 1:17) dt <- rbind(dt,dt)

mycode <-function() {
  test1 <- dt[,
              list(c(sales_ccy, cost_ccy),c(sum(sales_amt), sum(cost_amt))), # this will create two new columns with ccy and amt
              keyby=list(sales_ccy, cost_ccy) 
             ][,
                sum(V2), # this will aggregate the new columns
                by=V1
              ]
  rm(test1)
}

suggesteEdit <- function() {

  test2 <- dt[ , .(c(sales_ccy, cost_ccy), c(sales_amt, cost_amt)) # combine cols
   ][, .(tot_amt = sum(V2)), keyby= .(ccy = V1)          # aggregate + reorder
     ]
   rm(test2)
}

meltWithDataTable195 <- function() {
  test3 <- melt(dt, measure = list( c(1,3), c(2,4) ))[, .(tot_amt = sum(value2)), keyby = .(ccy=value1)]
  rm(test3)
}

microbenchmark(
  mycode(),
  suggesteEdit(),
  meltWithDataTable195()
)

Result

Unit: milliseconds
                   expr      min       lq     mean   median       uq      max neval
               mycode() 12.27895 12.47456 15.04098 12.80956 14.73432 45.26173   100
         suggesteEdit() 25.36581 29.56553 42.52952 33.39229 59.72346 69.74819   100
 meltWithDataTable195() 25.71558 30.97693 47.77700 58.68051 61.23996 66.49597   100

Answer 4

Edited Another way to do this using aggregate()

df = data.frame(ccy = c(dt$sales_ccy, dt$cost_ccy), total_amt = c(dt$sales_amt, dt$cost_amt))
out= aggregate(total_amt ~ ccy, data = df, sum)

Answer 5

Dirty but works

# Bind costs and sales
df <- rbind(df[,list(ccy = cost_ccy, total_amt = cost_amt)], 
            df[,list(ccy = sales_ccy, total_amt = sales_amt)])
# Sum for every currency
df[, sum(total_amt), by = ccy]
   ccy  V1
1: GBP 300
2: USD 700
3: EUR 600