累计和数据组织

Question

I have about 40000 values for rainfall data from different samples which will constantly be updated. The csv file is organised like this:

NAME;       YEAR;   ID;     VALUE
Sample1;    1998;   354;    45
Sample1;    1999;   354;    23
Sample1;    2000;   354;    66
Sample1;    2001;   354;    98
Sample1;    2002;   354;    36
Sample1;    2003;   354;    59
Sample1;    2004;   354;    64
Sample1;    2005;   354;    23
Sample1;    2006;   354;    69
Sample1;    2007;   354;    94
Sample1;    2008;   354;    24
Sample2;    1964;   1342;    7
Sample2;    1965;   1342;   24
Sample3;    2002;   859;    90
Sample3;    2003;   859;    93
Sample3;    2004;   859;    53
Sample3;    2005;   859;    98 

What I'd like to do with an R script is the following: Create a new row where for a group of samples (eg for all Sample1 and then start over at the value for all Sample2 and then start over at the value for all Sample3 and so on) are summed up based on the previous value (cumulative sum of rainfall data), for example for sample 1 results in a row like in this example CUM_RAINFALL (for first example something like this: 45 for CUM_RAINFALL 1 and then 45+23, and then 68+66, and then 134+232 and so on until the end of Sample1, the value of Sample2 should be taken over and the procedure should start all over again)

NAME;       YEAR;   ID;     VALUE    CUM_RAINFALL
Sample1;    1998;   354;    45;       45
Sample1;    1999;   354;    23;       68
Sample1;    2000;   354;    66;      134
Sample1;    2001;   354;    98;      232
Sample1;    2002;   354;    36;      268
Sample1;    2003;   354;    59;      327
Sample1;    2004;   354;    64;      391
Sample1;    2005;   354;    23;      414
Sample1;    2006;   354;    69;      483
Sample1;    2007;   354;    94;      577
Sample1;    2008;   354;    24;      601
Sample2;    1964;   1342;    7;      7
Sample2;    1965;   1342;   24;      31
Sample3;    2002;   859;    90;      90
Sample3;    2003;   859;    93;      183
Sample3;    2004;   859;    53;      236
Sample3;    2005;   859;    98;      334

From this I would like to write a new file containing all rows which have more than 3 values (in the given example Sample2 wouldn't be written into the file, because it contains only 2 values)

Is there an easy way to do this in R? Any help is appreciated! Under the following link you'll find a csv with the data: https://dl.dropboxusercontent.com/u/16277659/sample.cs

Answer 1

Here's a solution using data.table package assuming your data is stored in dat :

require(data.table)
ans = setDT(dat)[, crain := cumsum(VALUE[.N > 3L]), by=NAME][!is.na(crain)]

• setDT converts data.frame to data.table
• Then, we group by NAME and calculate, for each unique group, the cumulative sum of VALUE for that group only if the number of observations for that group (= .N , inbuilt special variable) is > 3L. And we assign the values to new column crain by reference.
• Since we did not compute cumsum for groups with <= 3L observations, they will have NA values in them. We exploit that to subset the desired result.

Now, you can use write.table(.) on ans , as shown in other answers.

Note: This answer assumes that your data set does not contain NA values for VALUE column of course.

Answer 2

40k observations should do fine in base R.

d$CUMRAIN <- unlist(by(d$VALUE, d$NAME, cumsum), use.names = FALSE)
d
#       NAME YEAR   ID VALUE CUMRAIN
# 1  Sample1 1998  354    45      45
# 2  Sample1 1999  354    23      68
# 3  Sample1 2000  354    66     134
# 4  Sample1 2001  354    98     232
# 5  Sample1 2002  354    36     268
# 6  Sample1 2003  354    59     327
# 7  Sample1 2004  354    64     391
# 8  Sample1 2005  354    23     414
# 9  Sample1 2006  354    69     483
# 10 Sample1 2007  354    94     577
# 11 Sample1 2008  354    24     601
# 12 Sample2 1964 1342     7       7
# 13 Sample2 1965 1342    24      31
# 14 Sample3 2002  859    90      90
# 15 Sample3 2003  859    93     183
# 16 Sample3 2004  859    53     236
# 17 Sample3 2005  859    98     334

I use by here, but here are some other ways to calculate the cumsum by factor level

mapply(cumsum, with(d, split(VALUE, NAME)))
sapply(unname(split(d$VALUE, d$NAME)), cumsum)
unsplit(sapply(split(d$VALUE, d$NAME), cumsum), d$NAME) 

The latter is probably the most favorable since it drops the factor names.

There's also

library(plyr)
ddply(d, .(NAME), mutate, CUMSUM = cumsum(VALUE))     

---

To subset for more than three observations, you can use a simple table

t <- table(d$NAME)
ss <- d[d$NAME %in% names(t)[t > 3], ]

Then to write it to file with

write.table(ss, "filename", sep = ";")

Answer 3

Here's another approach using dplyr

library(dplyr)

data %>%                                   # your data frame
  group_by(NAME) %>%                       # the grouping variable. could add more variables if necessary
  filter(n() > 3) %>%                      # n()  calculates the number of rows per group and then only those with more than 3 are filtered (selected)
  mutate(CUMRAIN = cumsum(VALUE)) %>%      # add a new column "CUMRAIN"
  write.table(., "test.csv", sep = ";")    # write the subset to a file. The "." indicates that it uses the output of the previous operations piped by %>%   

The operations are "piped" together using the %>% operator.

Update: as noted in @Arun's answer, it's not necessary to calculate the cumulative rain for those sample with less than 3 observations, so we can use the filter operation first (before mutate) to make a subset with all samples containing more than 3 observations and afterwards compute the cumulative rain.