使用R和dplyr执行累积组操作

Question

I'm trying to process data based on a sequential group id. There are J groups and I want to run the data processing function for groups i < j=1..J

The most trivial case is when each row is it's own group and you calculate the cumulative sum. However I have multiple rows in each group and the processing is more complicated than summation.

Here is an minimal example of my data format:

row | group | value
----|-------|------
  1 |     1 |  2065
  2 |     1 |  2075
  3 |     2 | 18008
  4 |     2 | 17655
  : |     : |     :
N-1 |   J-1 |  2345
  N |     J |  5432

One solution I've thought of is to replicate my data, stacking it and reassigning the groups in each data so that group i<j to j. This would result in a very long data frame like such:

row | group | value
----|-------|------
  1 |     1 |  2065
  2 |     1 |  2075
  3 |     2 |  2065
  4 |     2 |  2075
  5 |     2 | 18008
  6 |     2 | 17655
  : |     : |     :

However this seems tedious and inefficient as my data will be copied many times.

Does anyone know of a more efficient way of processing the data in a cumulative group by way?

Answer 1

Here are three example, one with aggregate , one with data.table and the last one with dplyr as you asked.

First create the dataframe

library(data.table)
library(dplyr)

group <- c(1,1,2,2,3)
value <- c(2065, 2075, 18008, 17655, 561)

With data.table you can use this function

dat <- data.table(group, value)
recap <- dat[, list(somma = sum(value)), by = group]

With aggregate from the package stats

dat <- data.frame(group, value)
aggregate(dat$value, by=list(Group=dat$group), FUN=sum)

Then with dplyr

dat %>%
    group_by(group) %>%
    summarise(result = sum(value))

These will give you

group | result
---------------
  1   |  4140
  2   |  35663
  3   |  561

Answer 2

One methodology that should work here is to split the data.frame by group id, and then run a for loop (or lapply ) with the the cumulative groups. Below is an example using a for loop as I think it is will be more straightforward to implement.

# split data.frame by group ID
myList <- split(df, df$group)
# initialize empty output list
myOutputList <- list()

# loop through group IDs, including the next one
for(i in seq_along(unique(df$group))) {
  # create temporary df for analysis
  myTempDf <- do.call(rbind, myList[seq_len(i)])

  ## perform analysis on myTempDf here ##

  # save results
  myOutputList[[i]] <- list(<list of analysis ouput>)
}

The output would be a nested list. I'd recommend naming each item in the nested list to make it easier to access, like myOutputList[[i]][["regression.1"]] .

Note that this assumes that the groups are properly sorted properly in the original data.frame and that the group ids are the counting numbers 1,2,3,4,... as in your example.

Answer 3

Here are several approaches:

1) sqldf This is being transferred from the comments. I had originally put it there since it is not a dplyr solution but it seems you are considering others. We join the unique group values with the data frame on the indicated condition. A single SQL statement will do it:

DF <- data.frame(group = c(1, 1, 2, 2), value = 1:4) # test data

library(sqldf)
outDF <- sqldf("select a.[group], b.value 
                from 
                     (select distinct [group] from DF) a 
                     join DF b on a.[group] >= b.[group]")

giving:

> outDF
  group value
1     1     1
2     1     2
3     2     1
4     2     2
5     2     3
6     2     4

and now we can process over the groups. Depending on what fun looks like one of these might do it:

aggregate(value ~ group, outDF, fun)

tapply(outDF$value, outDF$group, fun)

by(outDF, outDF$group, fun)

ave(outDF$value, outDF$group, FUN = fun)

If the operation were sum, say, rather than a separate aggregation it could be combined with the above like this.

sqldf("select a.[group], sum(b.value) cumsum
       from (select distinct [group] from DF) a join DF b on a.[group] >= b.[group] 
       group by a.[group]")

giving:

  group cumsum
1     1      3
2     2     10

Note that

• group is an SQL keyword which is why we escaped it using [group]

• we have assumed that it is desired to accumulate groups that are numerically equal or less than the current group which is the case in the example in the question. If a different order were desired we could create another grouping variable whose ordering reflected that desired.

2) base This does not use any packages. We have assumed that it is desired to accumulate the current group and groups that appear prior to it in the split so that groups are accumulated in numerical order; however, if we want a different order we could make group into a factor and order the levels as desired since split output will be in the order of the grouping factor's levels.

L <- Reduce(rbind, split(DF, DF$group), acc = TRUE)
do.call("rbind", lapply(L, transform, group = tail(group, 1)))

giving:

  group value
1     1     1
2     1     2
3     2     1
4     2     2
5     2     3
6     2     4

3) magrittr (2) can be rewritten using magrittr like this:

library(magrittr)

DF %>%
  split(.$group) %>%
  Reduce(f = rbind, acc = TRUE) %>%
  lapply(transform, group = tail(group, 1)) %>%
  do.call(what = "rbind")

giving the same result as in (2).