如何根据分位数按日期删除行？

Question

My problem is the following: I would like to remove rows in a data frame which are lower than the 50th percentile defined for each date. The following example illustrate my problem.

I have the following data frame:

date <- c("01.02.2011","01.02.2011","01.02.2011","01.02.2011","01.02.2011","01.02.2011",
          "01.02.2011","01.02.2011","01.02.2011","01.02.2011",
          "02.02.2011","02.02.2011","02.02.2011","02.02.2011","02.02.2011","02.02.2011",
          "02.02.2011","02.02.2011","02.02.2011","02.02.2011")
date <- as.Date(date, format="%d.%m.%Y")
ID <- c("A","B","C","D","E","F","G","H","I","J",
        "A","B","C","D","E","F","G","H","I","J")
values <- as.numeric(c("1","8","2","3","5","13","2","4","1","16",
                       "4","2","12","16","8","1","7","11","2","10"))

df <- data.frame(ID, date, values)

Looking like this:

   ID       date values
1   A 2011-02-01      1
2   B 2011-02-01      8
3   C 2011-02-01      2
4   D 2011-02-01      3
5   E 2011-02-01      5
6   F 2011-02-01     13
7   G 2011-02-01      2
8   H 2011-02-01      4
9   I 2011-02-01      1
10  J 2011-02-01     16
11  A 2011-02-02      4
12  B 2011-02-02      2
13  C 2011-02-02     12
14  D 2011-02-02     16
15  E 2011-02-02      8
16  F 2011-02-02      1
17  G 2011-02-02      7
18  H 2011-02-02     11
19  I 2011-02-02      2
20  J 2011-02-02     10

I would like to delete all the rows for each date where values are below the 50th percentile (defined by date) in order to obtain:

   ID       date values
2   B 2011-02-01      8
5   E 2011-02-01      5
6   F 2011-02-01     13
8   H 2011-02-01      4
10  J 2011-02-01     16
13  C 2011-02-02     12
14  D 2011-02-02     16
15  E 2011-02-02      8
18  H 2011-02-02     11
20  J 2011-02-02     10

If any editing of my question is needed, do not hesitate to let me know

Answer 1

You have several ways to do that. Some solutions here but there exists much more way to do that. They all apply the same idea: first compute median by date, then filter your data.

data.table

If you want to use data.table , first you update your data by reference using := then you filter. data.table is a very efficient approach if your dataset is voluminous.

library(data.table)
setDT(df)

df[, quant := quantile(values, probs = .5),by = "date"]
df2 <- df[values>quant]
df2[,'quant' := NULL]

df2
    ID       date values
 1:  B 2011-02-01      8
 2:  E 2011-02-01      5
 3:  F 2011-02-01     13
 4:  H 2011-02-01      4
 5:  J 2011-02-01     16
 6:  C 2011-02-02     12
 7:  D 2011-02-02     16
 8:  E 2011-02-02      8
 9:  H 2011-02-02     11
10:  J 2011-02-02     10

dplyr

With dplyr , you pipe your operations your operations: compute quantile by group and then filter

library(dplyr)
df %>%
   group_by(date) %>%
   mutate(quant = quantile(values, .5)) %>%
   filter(values>quant) %>%
   select(-quant)

Groups:   date [2]
   ID    date       values
   <fct> <date>      <dbl>
 1 B     2011-02-01      8
 2 E     2011-02-01      5
 3 F     2011-02-01     13
 4 H     2011-02-01      4
 5 J     2011-02-01     16
 6 C     2011-02-02     12
 7 D     2011-02-02     16
 8 E     2011-02-02      8
 9 H     2011-02-02     11
10 J     2011-02-02     10