[英]R nested foreach loop
我有一个输入数据集:
# environment
require(pacman)
p_load(
data.table
, doParallel
, foreach
)
doParallel::registerDoParallel(makeCluster(4))
# create input
runDT <- data.table(run = c(F,T,F,T)
, input1 = 1:4
, run_id = 1:4)
print(runDT)
run input1 run_id
1: FALSE 1 1
2: TRUE 2 2
3: FALSE 3 3
4: TRUE 4 4
这是另一个原始数据集:
dataDT <- data.table(
ID = 1:4
, c1 = c(1:4))
print(dataDT)
ID c1
1: 1 1
2: 2 2
3: 3 3
4: 4 4
我想运行嵌套的foreach循环,但这给了我一个错误:
# run
row_run <- runDT[run == T, run_id]
resultsDT <- foreach::foreach(
k = 1:length(row_run), .inorder = FALSE, .packages = c("data.table")) %dopar% {
# get the input for this run
inputDT <- runDT[run_id == row_run[k],]
# apply the input for all dataDT rows
result_run <- foreach::foreach(
j = 1:nrow(dataDT), .inorder = FALSE, .packages = c("data.table")) %dopar% {
dataDT_run <- dataDT[ID == j,]
dataDT_run[, c("o1", "run_id") := list(
c1 + inputDT[, input1]
, inputDT[, run_id]
)]
return(dataDT_run[, c("o1", "run_id"), with = FALSE])
}
result_run <- rbindlist(result_run)
return(result_run)
}
Error in { : task 1 failed - "could not find function "%dopar%""
resultsDT <- rbindlist(resultsDT)
print(resultsDT)
我期望看到的结果是:
resultsDT <- data.table(
o1 = c((1:4) + 2,c(1:4) + 4)
, run_id = c(rep(2,4),rep(4,4))
)
print(resultsDT)
o1 run_id
1: 3 2
2: 4 2
3: 5 2
4: 6 2
5: 5 4
6: 6 4
7: 7 4
8: 8 4
然后,我将第一个%dopar%
更改为%:%
,但是又出现了另一个错误:
Error in foreach::foreach(k = 1:length(row_run), .inorder = FALSE, .packages = c("data.table")) %:% :
no function to return from, jumping to top level
如何解决?
已修复。似乎必须将inputDT <- runDT[run_id == row_run[k],]
放入循环内:
resultsDT <- foreach::foreach(
k = 1:length(row_run), .inorder = FALSE, .packages = c("data.table"), .combine = 'rbind') %:%
# apply the input for all dataDT rows
foreach::foreach(
j = 1:nrow(dataDT), .combine = 'rbind') %dopar% {
# get the input for this run
inputDT <- runDT[run_id == row_run[k],]
dataDT_run <- dataDT[ID == j,]
dataDT_run[, c("o1", "run_id") := list(
c1 + inputDT[, input1]
, inputDT[, run_id]
)]
return(dataDT_run[, c("o1", "run_id"), with = FALSE])
}
print(resultsDT)
o1 run_id
1: 3 2
2: 4 2
3: 5 2
4: 6 2
5: 5 4
6: 6 4
7: 7 4
8: 8 4
但是,如果我们这样做, runDT
是否会被复制到RAM k * j次? 因为我的实际runDT
很大。
但是,如果我们这样做,runDT是否会被复制到RAM k * j次? 因为我的实际runDT很大。
我会回答您的其他问题
doParallel::registerDoParallel(makeCluster(4))
当您创建4个集群时,runDT将复制到您的4个集群中。
inputDT <- runDT[run_id == row_run[k],]
此外,假设k*j
为8,所有inputDT
大小均为100MB
。
size(Cluster1) : runDT + inputDT(100MB) + inputDT(100MB) + etc
size(Cluster2) : runDT + inputDT(100MB) + inputDT(100MB) + etc
size(Cluster3) : runDT + inputDT(100MB) + inputDT(100MB) + etc
size(Cluster4) : runDT + inputDT(100MB) + inputDT(100MB) + etc
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.