[英]Selecting tuning parameters with caret using standard deviation of custom metric
我正在使用帶有自定義擬合度量的插入符號,但我需要最大化不僅僅是這個度量,而是它的置信區間的下限。 所以我想最大化像mean(metric) - k * stddev(metric)
。 我知道如何手動執行此操作,但有沒有辦法告訴插入符號使用此功能自動選擇最佳參數?
是的,您可以通過“trainControl”對象的“summaryFunction”參數定義自己的選擇指標,然后使用train()
調用的“metric”參數。 關於這一點的詳細信息在Caret的模型調整頁面的“備用性能指標”部分中有詳細記錄: http : //caret.r-forge.r-project.org/training.html
我認為你沒有給任何人提供足夠的信息來准確寫出你正在尋找的內容,但這里有一個使用來自twoClassSummary函數的代碼的例子:
> library(caret)
> data(Titanic)
>
> #an example custom function
> roc <- function (data, lev = NULL, model = NULL) {
+ require(pROC)
+ if (!all(levels(data[, "pred"]) == levels(data[, "obs"])))
+ stop("levels of observed and predicted data do not match")
+ rocObject <- try(pROC:::roc(data$obs, data[, lev[1]]), silent = TRUE)
+ rocAUC <- if (class(rocObject)[1] == "try-error")
+ NA
+ else rocObject$auc
+ out <- c(rocAUC, sensitivity(data[, "pred"], data[, "obs"], lev[1]), specificity(data[, "pred"], data[, "obs"], lev[2]))
+ names(out) <- c("ROC", "Sens", "Spec")
+ out
+ }
>
> #your train control specs
> tc <- trainControl(method="cv",classProb=TRUE,summaryFunction=roc)
> #yoru model with selection metric specificed
> train(Survived~.,data=data.frame(Titanic),method="rf",trControl=tc,metric="ROC")
32 samples
4 predictors
2 classes: 'No', 'Yes'
No pre-processing
Resampling: Cross-Validation (10 fold)
Summary of sample sizes: 28, 29, 30, 30, 28, 28, ...
Resampling results across tuning parameters:
mtry ROC Sens Spec ROC SD Sens SD Spec SD
2 0.9 0.2 0.25 0.175 0.35 0.425
4 0.85 0.4 0.6 0.211 0.459 0.459
6 0.875 0.35 0.6 0.212 0.412 0.459
ROC was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.
在插入符號幫助列車功能方面有更多基本示例:
madSummary <- function (data,
lev = NULL,
model = NULL) {
out <- mad(data$obs - data$pred,
na.rm = TRUE)
names(out) <- "MAD"
out
}
robustControl <- trainControl(summaryFunction = madSummary)
marsGrid <- expand.grid(degree = 1, nprune = (1:10) * 2)
earthFit <- train(medv ~ .,
data = BostonHousing,
method = "earth",
tuneGrid = marsGrid,
metric = "MAD",
maximize = FALSE,
trControl = robustControl)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.