[英]R: How to bin efficiently using time-varying breakpoints?
I'm working with a large data frame of 14 million rows, containing columns month
, firmID
, and firmSize
.我正在处理一个包含 1400 万行的大型数据框,其中包含
month
、 firmID
和firmSize
列。 In a separate data frame I have monthly breakpoints (quintiles essentially) for firm size.在一个单独的数据框中,我有公司规模的每月断点(基本上是五分位数)。
My goal is to add a fourth column quintile
to the first data frame.我的目标是在第一个数据框中添加第四列
quintile
。 In this column I would have a number from 1 to 5 corresponding to the size quintile the firmSize
belongs to in that specific month.在此列中,我将有一个从 1 到 5 的数字,对应于该特定月份中公司大小所属的大小五分
firmSize
。
I have the following loop that does the job but has a runtime of several hundreds of hours.我有以下循环可以完成这项工作,但运行时间为数百小时。
for (i in 1:length(df$month)) {
for (j in 1:4) {
if (df$size[i] <= breakpoints[which(df$month[i] == breakpoints$month),(j+1)]) {
df$quintile[i] <- j
break()
}
else {
df$quintile[i] <- 5
}
}
}
I have quite limited knowledge of eg the applications of dplyr and I was wondering if anyone has an idea about how to approach this problem so that I don't have to keep my laptop running for weeks.我对例如 dplyr 的应用程序的了解非常有限,我想知道是否有人知道如何解决这个问题,这样我就不必让我的笔记本电脑运行数周。
Thank you in advance!先感谢您!
Edit: Example data for the data frames: (thank you Ricardo for your suggestion!)编辑:数据框的示例数据:(感谢里卡多的建议!)
df df
month firmID firmSize
201001 46603210 9738635
201001 72913210 1166077
201001 00621210 3884422
201512 75991610 2932127
201512 45383610 1241272
201512 05766520 1931038
breakpoints断点
month Q1 Q2 Q3 Q4 Q5
201001 322770 1038300 2112300 4597580 28919700
201512 379340 1239800 2840630 7785700 46209140
I wonder if using findInterval
and data.table
might be worth pursuing and faster.我想知道使用
findInterval
和data.table
是否值得追求和更快。 This was adapted from this answer which I thought was very helpful.这是改编自我认为非常有帮助的这个答案。
findInterval
finds the index of one vector in another (assuming the other is non-decreasing). findInterval
在另一个向量中找到一个向量的索引(假设另一个向量是非递减的)。 In this case, breakpoints
columns from Q1
to Q5
forms the second vector, and the function will return the index based on the firmSize
value in the first data frame.在这种情况下,
breakpoints
列从Q1
到Q5
forms 第二个向量,function 将根据第一个数据帧中的firmSize
值返回索引。
library(data.table)
setDT(df)
setkey(df, month)
setDT(breakpoints)
setkey(breakpoints, month)
df[, quintile := findInterval(firmSize, breakpoints[.BY][, Q1:Q5]) + 1, by = month][]
Output Output
month firmID firmSize quintile
1: 201001 46603210 9738635 5
2: 201001 72913210 1166077 3
3: 201001 621210 3884422 4
4: 201512 75991610 2932127 4
5: 201512 45383610 1241272 3
6: 201512 5766520 1931038 3
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.