如何在MATLAB中选择RELIEFF算法中的k值

Question

I'm using relieff algorithm to investigate the ranking of various inputs for solving a classification problem. I have five inputs and about 500 observations. I'm using MATLAB to solve this.

I start off by setting the k nearest neighbors for the algorithm to 2 and vary it all the way till 450. The ranking computed for these vary wildly initially and then stabilizes as the size of k nearest neighbors approaches 150. I'm attaching a graph of weight (higher weight - higher ranking) of each of the five attributes vs the number of k nearest neighbors below.

I'm wondering how to choose the value of k given for the ReliefF algorithm ?

Answer 1

With the K vs Weights plot you've just answered your own question. That is indeed very smart.

The optimal K value for your dataset is where the elbow is (circa 350).
What does it mean? It basically means that taking into account another neighbour does not give a better modelling of the data.
You can object then that choosing 350 or 400 will lead to the same results since the weights are equal. Correct. However it is always recommended to choose for the smallest value because the model you're training will have a minor complexity (fewer number of neighbours to take into account) with respect to the same results (ie weights).

Such bruteforcing techniques are commonly used for many algorithms in machine learning:

• in K-NN to find the optimal number of neighbours
• in K-Means to find the optimal number of clusters
• in SVMs to find optimal tuning parameters

and so on and so forth...

I've been doing the very same experiment as you did, but with another dataset and I obtained the following plot:

and as you can see for this dataset the optimal K is circa 50.