[英]In R, how do I create individual time series (to loop trough and run a Mann-Kendall test) from a larger dataset?
[英]Speed-up a parallel process calculating a mann-kendall test over a huge dataset in R
假設每個月的時間步長都有一個龐大的氣候數據集,用於世界上的許多點。 然后將數據集定型為類型為data.frame
的數據:
lon,lat,data_month_1_yr_1,...,data_month_12_yr_100
例:
set.seed(123)
data<- data.frame(cbind(runif(10000,-180,180), runif(10000,-90,90))
, replicate(1200, runif(10000,0,150)))
我想在每個空間點的每月時間序列上執行Mann-Kendall測試(使用trend::mk.test
),並在data.frame
獲取主要統計信息。 為了加快這個非常漫長的過程,我並行化了我的代碼並編寫了如下內容:
coords<-data[,1:2] #get the coordinates out of the initial dataset
names(coords)<-c("lon","lat")
data_t<- as.data.frame(t(data[,3:1202])) #each column is now the time series associated to a point
data_t$month<-rep(seq(1,12,1),100) # month index as last column of the data frame
# start the parallel processing
library(foreach)
library(doParallel)
cores=detectCores() #count cores
cl <- makeCluster(cores[1]-1) #take all the cores minus 1 not to overload the pc
registerDoParallel(cl)
mk_out<- foreach(m=1:12, .combine = rbind) %:%
foreach (a =1:10000, .combine = rbind) %dopar% {
data_m<-data_t[which(data_t$month==m),]
library(trend) #need to load this all the times otherwise I get an error (don't know why)
test<-mk.test(data_m[,a])
mk_out_temp <- data.frame("lon"=coords[a,1],
"lat"=coords[a,2],
"p.value" = as.numeric(test$p.value),
"z_stat" = as.numeric(test$statistic),
"tau" = as.numeric(test$estimates[3]),
"month"= as.numeric(m))
mk_out_temp
}
stopCluster(cl)
head(mk_out)
lon lat p.value z_stat tau month
1 -76.47209 -34.09350 0.57759040 -0.5569078 -0.03797980 1
2 103.78985 -31.58639 0.64436238 0.4616081 0.03151515 1
3 -32.76831 66.64575 0.11793238 1.5635113 0.10626263 1
4 137.88627 -30.83872 0.79096910 0.2650524 0.01818182 1
5 158.56822 -67.37378 0.09595919 -1.6647673 -0.11313131 1
6 -163.59966 -25.88014 0.82325630 0.2233588 0.01535354 1
這運行得很好,並且給我確切的信息:一個矩陣,報告每個坐標和月份組合的MK統計信息。 盡管該過程是並行的,但是計算仍需要花費大量時間。
有沒有辦法加快這個過程? apply
家庭有使用職能的空間嗎?
您注意到您已經解決了問題。 可通過以下步驟之一獲得:
1:使用.packages
和.export
將必要的對象復制到foreach循環中。 這樣可以確保在嘗試訪問同一內存時每個實例不會沖突。
2:利用諸如data.table的tidyverse之類的高性能庫來執行子集和計算。
后者稍微復雜一點,但極大地提高了我的微型筆記本電腦的性能。 (對整個數據集執行所有計算大約需要1.5分鍾。)
以下是我添加的代碼。 請注意,我用並行包中的單個parLapply函數替換了foreach。
set.seed(123)
data<- data.frame(cbind(runif(10000,-180,180), runif(10000,-90,90))
, replicate(1200, runif(10000,0,150)))
coords<-data[,1:2] #get the coordinates out of the initial dataset
names(coords)<-c("lon","lat")
data_t<- as.data.frame(t(data[,3:1202])) #each column is now the time series associated to a point
data_t$month<-rep(seq(1,12,1),100) # month index as last column of the data frame
# start the parallel processing
library(data.table)
library(parallel)
library(trend)
setDT(data_t)
setDT(coords)
cores=detectCores() #count cores
cl <- makeCluster(cores[1]-1) #take all the cores minus 1 not to overload the pc
#user system elapsed
#17.80 35.12 98.72
system.time({
test <- data_t[,parLapply(cl,
.SD, function(x){
(
unlist(
trend::mk.test(x)[c("p.value","statistic","estimates")]
)
)
}
), by = month] #Perform the calculations across each month
#create a column that indicates what each row is measuring
rows <- rep(c("p.value","statistic.z","estimates.S","estimates.var","estimates.tau"),12)
final_tests <- dcast( #Cast the melted structure to a nice form
melt(cbind(test,rowname = rows), #Melt the data for a better structure
id.vars = c("rowname","month"), #Grouping variables
measure.vars = paste0("V",seq.int(1,10000))), #variable names
month + variable ~ rowname, #LHS groups the data along rows, RHS decides the value columns
value.var = "value", #Which column contain values?
drop = TRUE) #should we drop unused columns? (doesnt matter here)
#rename the columns as desired
names(final_tests) <- c("month","variable","S","tau","var","p.value","z_stat")
#finally add the coordinates
final_tests <- cbind(final_form,coords)
})
最后,可以通過用lapply
函數替換第二個循環來輕松解決此問題 (受此答案啟發)。 現在,執行時間僅為幾秒鍾。 向量化仍然是解決R中執行時間的最佳解決方案(請參閱此職位和this )
我在下面共享最終代碼以供參考:
set.seed(123)
data<- data.frame(cbind(runif(10000,-180,180), runif(10000,-90,90)), replicate(1200, runif(10000,0,150)))
coords<-data[,1:2]
names(coords)<-c("lon","lat")
data_t<- as.data.frame(t(data[,3:1202]))
data_t$month<-rep(seq(1,12,1),100)
library(foreach)
library(doParallel)
cores=detectCores()
cl <- makeCluster(cores[1]-1) #take all the cores minus 1
registerDoParallel(cl)
mk_out<- foreach(m=1:12, .combine = rbind) %dopar% {
data_m<-data_t[which(data_t$month==m),]
library(trend)
mk_out_temp <- do.call(rbind,lapply(data_m[1:100],function(x)unlist(mk.test(x))))
mk_out_temp <-cbind(coords,mk_out_temp,rep(m,dim(coords)[1]))
mk_out_temp
}
stopCluster(cl)
head(mk_out)
head(mk_out)
lon lat data.name p.value statistic.z null.value.S parameter.n estimates.S estimates.varS
1 -76.47209 -34.09350 x 0.577590398263635 -0.556907839290681 0 100 -188 112750
2 103.78985 -31.58639 x 0.644362383361713 0.461608102085858 0 100 156 112750
3 -32.76831 66.64575 x 0.117932376736468 1.56351131351662 0 100 526 112750
4 137.88627 -30.83872 x 0.79096910003836 0.265052394100912 0 100 90 112750
5 158.56822 -67.37378 x 0.0959591933285242 -1.66476728429674 0 100 -560 112750
6 -163.59966 -25.88014 x 0.823256299016955 0.223358759073802 0 100 76 112750
estimates.tau alternative method pvalg rep(m, dim(coords)[1])
1 -0.037979797979798 two.sided Mann-Kendall trend test 0.577590398263635 1
2 0.0315151515151515 two.sided Mann-Kendall trend test 0.644362383361713 1
3 0.106262626262626 two.sided Mann-Kendall trend test 0.117932376736468 1
4 0.0181818181818182 two.sided Mann-Kendall trend test 0.79096910003836 1
5 -0.113131313131313 two.sided Mann-Kendall trend test 0.0959591933285242 1
6 0.0153535353535354 two.sided Mann-Kendall trend test 0.823256299016955 1
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.