相当于 R 中插入符号模型 selectionFunction 的 mlr

Question

The caret library in R has a hyper-parameter 'selectionFunction' inside trainControl(). It's used to prevent over-fitting models using Breiman's one standard error rule, or tolerance, etc.

Does mlr have an equivalent? If so, which function is it within?

Answer 1

mlr only allows to choose optimal hyperparameters by optimizing certain measures/metrics.

However, essentially each "measure" in mlr is just a function that specifies how a certain performance is handled. You can try to write your own custom measure as outlined in this vignette .

Other than that, it might be worth opening this as a feature request in the new mlr3 framework, specifically in mlr3measures , since mlr itself is deprecated.

Answer 2

Posting an answer to my own question, I found this. .

Estimate relative overfitting.

Source: R/relativeOverfitting.R

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

estimateRelativeOverfitting(
  predish,
  measures,
  task,
  learner = NULL,
  pred.train = NULL,
  iter = 1
)

Arguments

• predish - ( ResampleDesc ResamplePrediction Prediction ) Resampling strategy or resampling prediction or test predictions.
• measures - ( Measure list of Measure ) Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure .
• task - ( Task ) The task.
• learner - ( Learner character(1) ) The learner. If you pass a string the learner will be created via makeLearner .
• pred.train - ( Prediction ) Training predictions. Only needed if test predictions are passed.
• iter - ( integer ) Iteration number. Default 1, usually you don't need to specify this. Only needed if test predictions are passed.