[英]Moving values between rows without a for loop in R
I have written some code used to organize data sampled at different frequencies, but I made extensive use of for-loops, which slow the code's operation down significantly when the data set is large. 我编写了一些用于组织以不同频率采样的数据的代码,但是我广泛使用了for循环,当数据集很大时,这会大大降低代码的操作速度。 I've been going through my code, finding ways to remove for-loops to speed it up, but one of the loops has got me stumped. 我一直在浏览我的代码,找到删除for循环以加快速度的方法,但其中一个循环让我感到难过。
As an example, let's say the data was sampled at 3Hz, so I get three rows for every second of data. 举个例子,假设数据是以3Hz采样的,所以每秒数据都会得到三行。 However, the variables A, B, and C are sampled at 1Hz each, so I will get one value every three rows for each of them. 但是,变量A,B和C各自以1Hz采样,因此每三行我将获得一个值。 The variables are sampled consecutively within the one second period, resulting in a diagonal nature to the data. 在一秒周期内连续采样变量,导致数据的对角性质。
To further complicate things, sometimes a row is lost in the original data set. 更复杂的是,有时在原始数据集中会丢失一行。
My goal is this: Having identified the rows that I wish to keep, I want to move the non-NA values from the subsequent rows up into the keeper rows. 我的目标是:确定了我希望保留的行后,我想将后续行中的非NA值移动到守护者行中。 If it weren't for the lost data issue, I would always keep the row containing a value for the first variable, but if one of these rows is lost, I will be keeping the next row. 如果它不是丢失的数据问题,我将始终保持行包含第一个变量的值,但如果其中一行丢失,我将保留下一行。
In the example below, the sixth sample and the tenth sample are lost. 在下面的示例中,第六个样本和第十个样本丢失。
A <- c(1, NA, NA, 4, NA, 7, NA, NA, NA, NA)
B <- c(NA, 2, NA, NA, 5, NA, 8, NA, 11, NA)
C <- c(NA, NA, 3, NA, NA, NA, NA, 9, NA, 12)
test_df <- data.frame(A = A, B = B, C = C)
test_df
A B C
1 1 NA NA
2 NA 2 NA
3 NA NA 3
4 4 NA NA
5 NA 5 NA
6 7 NA NA
7 NA 8 NA
8 NA NA 9
9 NA 11 NA
10 NA NA 12
keep_rows <- c(1, 4, 6, 9)
After I move the values up into the keeper rows, I will remove the interim rows, resulting in the following: 将值移动到守护者行后,我将删除临时行,从而产生以下结果:
test_df <- test_df[keep_rows, ]
test_df
A B C
1 1 2 3
2 4 5 NA
3 7 8 9
4 NA 11 12
In the end, I only want one row for each second of data, and NA values should only remain where a row of the original data was lost. 最后,我只希望每秒数据有一行,而NA值只应保留原始数据行丢失的位置。
Does anyone have any ideas of how to move the data up without using a for-loop? 有没有人有任何关于如何在不使用for循环的情况下移动数据的想法? I'd appreciate any help! 我很感激任何帮助! Sorry if this question is too wordy; 对不起,如果这个问题太罗嗦了; I wanted to err on the side of too much information rather than not enough. 我想在太多信息方面犯错,而不是不够。
This should do it: 这应该这样做:
test_df = with(test_df, cbind(A[1:(length(A)-2)], B[2:(length(B)-1)], C[3:length(C)]))
test_df = data.frame(test_df[!apply(test_df, 1, function(x) all(is.na(x))), ])
colnames(test_df) = c('A', 'B', 'C')
> test_df
A B C
1 1 2 3
2 4 5 NA
3 7 8 9
4 NA 11 12
And if you want something even faster : 如果你想要更快的东西:
test_df = data.frame(test_df[rowSums(is.na(test_df)) != ncol(test_df), ])
Building on the great answer by @John Colby, we can get rid of the apply step and speed it up quite a bit (about 20x): 在@John Colby的伟大答案的基础上,我们可以摆脱应用步骤并加速相当多(大约20倍):
# Create a bigger test set
A <- c(1, NA, NA, 4, NA, 7, NA, NA, NA, NA)
B <- c(NA, 2, NA, NA, 5, NA, 8, NA, 11, NA)
C <- c(NA, NA, 3, NA, NA, NA, NA, 9, NA, 12)
n=1e6; test_df = data.frame(A=rep(A, len=n), B=rep(B, len=n), C=rep(C, len=n))
# John Colby's method, 9.66 secs
system.time({
df1 = with(test_df, cbind(A[1:(length(A)-2)], B[2:(length(B)-1)], C[3:length(C)]))
df1 = data.frame(df1[!apply(df1, 1, function(x) all(is.na(x))), ])
colnames(df1) = c('A', 'B', 'C')
})
# My method, 0.48 secs
system.time({
df2 = with(test_df, data.frame(A=A[1:(length(A)-2)], B=B[2:(length(B)-1)], C=C[3:length(C)]))
df2 = df2[is.finite(with(df2, A|B|C)),]
row.names(df2) <- NULL
})
identical(df1, df2) # TRUE
...The trick here is that A|B|C
is only NA
if all values are NA
. ...这里的技巧是, A|B|C
是唯一的NA
,如果所有的值都是NA
。 This turns out to be much faster than calling all(is.na(x))
on each row of a matrix using apply
. 这比使用apply
在矩阵的每一行上调用all(is.na(x))
要快得多。
EDIT @John has a different approach that also speeds it up. 编辑 @John有一个不同的方法,也加快了它。 I added some code to turn the result into a data.frame with correct names and timed it. 我添加了一些代码将结果转换为具有正确名称的data.frame并定时。 It seems to be pretty much the same speed as my solution. 它似乎与我的解决方案速度几乎相同。
# John's method, 0.50 secs
system.time({
test_m = with(test_df, cbind(A[1:(length(A)-2)], B[2:(length(B)-1)], C[3:length(C)]))
test_m[is.na(test_m)] <- -1
test_m <- test_m[rowSums(test_m) > -3,]
test_m[test_m == -1] <- NA
df3 <- data.frame(test_m)
colnames(df3) = c('A', 'B', 'C')
})
identical(df1, df3) # TRUE
EDIT AGAIN ...and @John Colby's updated answer is even faster! 再次编辑 ...... @John Colby的更新答案更快!
# John Colby's method, 0.39 secs
system.time({
df4 = with(test_df, cbind(A[1:(length(A)-2)], B[2:(length(B)-1)], C[3:length(C)]))
df4 = data.frame(df4[rowSums(is.na(df4)) != ncol(df4), ])
colnames(df4) = c('A', 'B', 'C')
})
identical(df1, df4) # TRUE
So your question was just about moving up without a loop. 所以你的问题只是在没有循环的情况下向上移动。 So apparently you solved the first step already. 显然你已经解决了第一步。
> test_m <- with( test_df, cbind(A[1:(length(A)-2)], B[2:(length(B)-1)], C[3:length(C)]) )
> test_m
[,1] [,2] [,3]
[1,] 1 2 3
[2,] NA NA NA
[3,] NA NA NA
[4,] 4 5 NA
[5,] NA NA NA
[6,] 7 8 9
[7,] NA NA NA
[8,] NA 11 12
Which is now a matrix. 现在是一个矩阵。 You can easily eliminate the rows for which there is no data point now without a loop. 在没有循环的情况下,您可以轻松地消除现在没有数据点的行。 If you want it back to a data.frame then you could use a different method but this one will run the fastest for a large mount of data. 如果您希望它返回到data.frame,那么您可以使用不同的方法,但是这个方法对于大量数据运行速度最快。 I like to just make the NA's an impossible value... perhaps -1 but you'll know best for your data... perhaps -pi. 我想让NA成为一个不可能的价值......也许-1但你知道你的数据最好......也许-pi。
test_m[is.na(test_m)] <- -1
And now just select the rows for a property of those impossible numbers 现在只需为那些不可能数字的属性选择行
test_m <- test_m[rowSums(test_m) > -3,]
And, if you want you can put the NA's back. 并且,如果你想,你可以把NA放回去。
test_m[test_m == -1] <- NA
test_m
[,1] [,2] [,3]
[1,] 1 2 3
[2,] 4 5 NA
[3,] 7 8 9
[4,] NA 11 12
There's no loop ( for
or apply
) and the one function applied across rows of the matrix is specially optimized and runs very fast (rowSums). 没有循环( for
或apply
),并且跨矩阵行应用的一个函数经过特殊优化并且运行速度非常快(rowSums)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.