[英]subset recursively a data.frame
I have a data frame with close to a 4 million of rows in it. 我有一个包含近400万行的数据框。 I need an efficient to way to subset the data based on two criteria.
我需要一种有效的方法来基于两个条件对数据进行子集化。 I can do this is a for loop but was wondering if there is a more elegant way to do this, and obviously more efficient.
我可以这样做是一个for循环,但是想知道是否有更优雅的方法来执行此操作,并且效率显然更高。 The data.frame looks like this:
data.frame看起来像这样:
SNP CHR BP P
rs1000000 chr1 126890980 0.000007
rs10000010 chr4 21618674 0.262098
rs10000012 chr4 1357325 0.344192
rs10000013 chr4 37225069 0.726325
rs10000017 chr4 84778125 0.204275
rs10000023 chr4 95733906 0.701778
rs10000029 chr4 138685624 0.260899
rs1000002 chr3 183635768 0.779574
rs10000030 chr4 103374154 0.964166
rs10000033 chr2 139599898 0.111846
rs10000036 chr4 139219262 0.564791
rs10000037 chr4 38924330 0.392908
rs10000038 chr4 189176035 0.971481
rs1000003 chr3 98342907 0.000004
rs10000041 chr3 165621955 0.573376
rs10000042 chr3 5237152 0.834206
rs10000056 chr4 189321617 0.268479
rs1000005 chr1 34433051 0.764046
rs10000062 chr4 5254744 0.238011
rs10000064 chr4 127809621 0.000044
rs10000068 chr2 36924287 0.000003
rs10000075 chr4 179488911 0.100225
rs10000076 chr4 183288360 0.962476
rs1000007 chr2 237752054 0.594928
rs10000081 chr1 17348363 0.517486
rs10000082 chr1 167310192 0.261577
rs10000088 chr1 182605350 0.649975
rs10000092 chr4 21895517 0.000005
rs10000100 chr4 19510493 0.296693
The first I need to do is to select those SNP
with a P
value lower than a threshold, then order this subset by CHR
and POS
. 我首先要做的是选择那些
P
值低于阈值的SNP
,然后按CHR
和POS
排序此子集。 This is the easy part, using subset
and order
. 这是简单的部分,使用
subset
和order
。 However, the next step is the tricky one. 但是,下一步是棘手的。 Once I have this subset, I need to fetch all the
SNP
that fall into a 500,000 window up and down from the significant SNP
, this step will define a region. 一旦我有这个子集,我需要获取所有
SNP
从显著陷入500,000窗口上下SNP
,这一步将定义一个区域。 I need to do it for all the significant SNP
and store each region into a list or something similar to carry out further analysis. 我需要对所有重要的
SNP
都执行此操作,并将每个区域存储到列表或类似内容中以进行进一步的分析。 For example, in the displayed data frame the most significant SNP
(ie below a threshold of 0.001) for CHR==chr1
is rs1000000
and for CHR==chr4
is rs10000092
. 例如,在显示的数据帧中,
CHR==chr1
的最高有效SNP
(即低于0.001的阈值)为rs1000000
,而CHR==chr4
的最高有效SNP
为rs10000092
。 Thus these two SNP
would define two regions and I need to fetch in each of these regions the SNPs that fall into a region of 500,000 up and down from the POS
of each of the most significant SNP
. 因此,这两个
SNP
将定义两个区域,我需要在每个这些区域中获取从每个最高SNP
的POS
上下掉入500,000的SNP
。
I know it a bit complicated, right now, I am doing the tricky part by hand but it takes a long time to do it. 我知道这有点复杂,现在,我正在手工完成棘手的部分,但是要花很长时间。 Any help would be appreciated.
任何帮助,将不胜感激。
Here is a partial solution ir R using data.table
, which is probably the fastest way to go in R when dealing with large datasets. 这是使用
data.table
R的部分解决方案,这可能是处理大型数据集时进入R的最快方法。
library(data.table) # v1.9.7 (devel version)
df <- fread("C:/folderpath/data.csv") # load your data
setDT(df) # convert your dataset into data.table
# Filter data under threshold 0.05 and Sort by CHR, POS
df <- df[ P < 0.05, ][order(CHR, POS)]
df[, {idx = (1:.N)[which.min(P)]
SNP[seq(max(1, idx - 5e5), min(.N, idx + 5e5))]}, by = CHR]
df[, fwrite(copy(.SD)[, SNP := SNP], paste0("output", SNP,".csv")), by = SNP]
ps. ps。 note that this answer uses
fwrite
, which is still in the development version of data.table
. 请注意,此答案使用
fwrite
,它仍在data.table
的开发版本中。 Go here for install instructions . 请转到此处获取安装说明 。 You could simply use
write.csv
, however you're dealing with a big dataset so speed is quite important and fwrite
is certainly one of the fastest alternatives . 您可以简单地使用
write.csv
,但是您要处理的是大型数据集,因此速度非常重要,而fwrite
无疑是最快的替代方法之一 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.