[英]Optimizing counting the number of unique values of one variable by another with data.table
我试图找到由变量/键y定义的每个组的一个变量x的唯一值的出现次数。
我一直在使用以下代码:
DT[,length(unique(x)),by=y] -> x_count_per_y
这可行,但是有点慢。 有没有一种方法可以针对data.table对此进行优化,还是我应该期望的最快?
使用uniqueN
1.9.5版本中的uniqueN。
在1.9.4中使用
uniqueN <- function(x) length(attr(data.table:::forderv(x, retGrp=TRUE),"starts",TRUE))
以编程方式使用它
byvar = "y"
countvar = "x"
DT[, uniqueN(.SD), by=byvar, .SDcols=countvar]
计时如下:
library(data.table)
library(microbenchmark)
N <- 1e6
DT <- data.table(x = sample(1e5,N,TRUE), y = sample(1e2,N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: milliseconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 85.58602 85.58602 85.58602 85.58602 85.58602 85.58602 1
# DT[, uniqueN(x), y] 92.71877 92.71877 92.71877 92.71877 92.71877 92.71877 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 97.51024 97.51024 97.51024 97.51024 97.51024 97.51024 1
N <- 1e7
DT <- data.table(x = sample(1e5,N,TRUE), y = sample(1e2,N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: milliseconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 1642.5212 1642.5212 1642.5212 1642.5212 1642.5212 1642.5212 1
# DT[, uniqueN(x), y] 843.0670 843.0670 843.0670 843.0670 843.0670 843.0670 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 804.7881 804.7881 804.7881 804.7881 804.7881 804.7881 1
N <- 1e7
DT <- data.table(x = sample(1e6,N,TRUE), y = sample(1e5,N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: seconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 3.025365 3.025365 3.025365 3.025365 3.025365 3.025365 1
# DT[, uniqueN(x), y] 4.734323 4.734323 4.734323 4.734323 4.734323 4.734323 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 5.905721 5.905721 5.905721 5.905721 5.905721 5.905721 1
N <- 1e7
DT <- data.table(x = sample(1e3,N,TRUE), y = sample(1e5,N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: seconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 2.906589 2.906589 2.906589 2.906589 2.906589 2.906589 1
# DT[, uniqueN(x), y] 4.731925 4.731925 4.731925 4.731925 4.731925 4.731925 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 7.084020 7.084020 7.084020 7.084020 7.084020 7.084020 1
N <- 1e7
DT <- data.table(x = sample(1e6,N,TRUE), y = sample(1e2,N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: milliseconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 1331.244 1331.244 1331.244 1331.244 1331.244 1331.244 1
# DT[, uniqueN(x), y] 998.040 998.040 998.040 998.040 998.040 998.040 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 1096.867 1096.867 1096.867 1096.867 1096.867 1096.867 1
很大程度上取决于数据,但是我填补了一个问题以了解这些时间安排。 角色的另一种:
N <- 1e7
DT <- data.table(x = sample(letters,N,TRUE), y = sample(letters[1:10],N,TRUE))
microbenchmark(times=1L,
DT[, length(unique(x)),y],
DT[, uniqueN(x),y],
DT[, uniqueN(.SD), by="y", .SDcols="x"])
# Unit: milliseconds
# expr min lq mean median uq max neval
# DT[, length(unique(x)), y] 1304.4865 1304.4865 1304.4865 1304.4865 1304.4865 1304.4865 1
# DT[, uniqueN(x), y] 573.8628 573.8628 573.8628 573.8628 573.8628 573.8628 1
# DT[, uniqueN(.SD), by = "y", .SDcols = "x"] 528.3269 528.3269 528.3269 528.3269 528.3269 528.3269 1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.