如何使用 R 在 K 近邻中找到 K 的最佳值？

Question

My dataset contains 5851 observations, and is split into a train (3511 observations) and test (2340 observations) set. I now want to train a model using KNN, with two variables. I want to do 10-fold CV, repeated 5 times, using ROC metric and the one-standard error rule and the variables are preprocessed. The code is shown below.

set.seed(44780)
ctrl_repcvSE <- trainControl(method = "repeatedcv", number = 10, repeats = 5,
                           summaryFunction = twoClassSummary, classProbs = TRUE,
                           selectionFunction = "oneSE")
tune_grid <- expand.grid(k = 45:75)
mod4 <- train(purchased ~ total_policies + total_contrib,
              data = mhomes_train, method = "knn",
              trControl= ctrl_repcvSE, metric = "ROC",
              tuneGrid = tune_grid, preProcess = c("center", "scale"))

The problem I have is that I already have tried so many different values of K (eg, K = 10:20, 30:40, 50:60, 150:160 + different tuning lengths. However, every time the output says that the chosen value for K is the one which is last, so for example for values of K = 70:80, the chosen value for K = 80, every time I do this. This means I should look further, because if the chosen value is K in that case then there are better values of K available which are above 80. How should I eventually find this one?

The assignment only specifies: For k-nearest neighbours, explore reasonable values of k using the total_policies and total_contrib variables only.

Answer 1

Welcome to Stack Overflow. Your question isn't easy to answer.

For k-nearest neighbours I use another function knn3 part of the caret library.

I'll give an example using the iris dataset. We try to get the accuracy of our model for different values for k and plot those accuracies.

library(data.table)
library(tidyverse)
library(scales)
library(caret)

dt <- as.data.table(iris)

# converting and scaling data ----
dt$Species      <- dt$Species %>% as.factor()
dt$Sepal.Length <- dt$Sepal.Length %>% scale()
dt$Sepal.Width  <-  dt$Sepal.Width %>% scale()
dt$Petal.Length <- dt$Petal.Length %>% scale()
dt$Petal.Width  <-  dt$Petal.Width %>% scale()

# remove in the real run ----
set.seed(1234567)

# split data into train and test - 3:1 ----
train_index <- createDataPartition(dt$Species, p = 0.75, list = FALSE)
train <- dt[train_index, ]
test <- dt[-train_index, ]

# values to check for k ----
K_VALUES  <- 20:1
test_acc  <- numeric(0)
train_acc <- numeric(0)

# calculate different models for each value of k ----
for (x in K_VALUES){
  model <- knn3(Species ~ ., data = train, k = x)
  pred_test <- predict(model, test, type = "class")
  pred_test_acc <- confusionMatrix(table(pred_test,
                                         test$Species))$overall["Accuracy"]
  test_acc <- c(test_acc, pred_test_acc)

  pred_train <- predict(model, train, type = "class")
  pred_train_acc <- confusionMatrix(table(pred_train,
                                          train$Species))$overall["Accuracy"]
  train_acc <- c(train_acc, pred_train_acc)
}

data <- data.table(x = K_VALUES, train = train_acc, test = test_acc)

# plot a validation curve ----
plot_data <- gather(data, "type", "value", -x)
g <- qplot(x = x,
           y = value,
           data = plot_data,
           color = type,
           geom = "path",
           xlim = c(max(K_VALUES),min(K_VALUES)-1))
print(g)

Now find a k with a good accuracy for your test data. That's the value you're looking for.

Disclosure: That's simplified but this approach should help you solving your problem.