Proper way to write the cost function in SVM in Matlab - unable to understand 'Cost' matrix

Question

I want to apply SVM to an imbalanced dataset and answer1 , answer2 suggest that it is possible to do so by tuning the parameters of the fitcsvm function. ALthough, SVM may not be a good choice for imbalanced data, yet I want to see the result for educational purpose.

How do I tune the parameters in SVM to put greater penalty to misclassification error for the true class (labelled as 1 ) since my data is naturally imbalanced, having less number of 1 's in contrast to 0 (false). Only 2% are labelled as 1 .

• The data set has 1473 samples (98%) labeled as 0 and 27 samples (2%) as 1 .

• The training data has 1000 samples labeled as 0 and 12 samples as 1 .

• The test data has 473 samples (97%) as 0 and 15 samples (3%) as 1 . I applied twice penalty to 1 by using a cost matrix, c as below:

 c=[0 2.2;1 0]; model = fitcsvm(train_x,train_y,'KernelFunction', 'rbf', 'Cost',c); [predLabel,score] = predict(model,test_x); 

The result is

Precision for label 0: 9.692623e-01
Precision for label 1: NaN
Recall for label 0: 1
Recall for label 1: 0

Accuracy = 96.9%
Average err = 0.03

The confusion matrix is

    473     0
    15     0

The answer in the predict vector are all 1 labels. Obviously, the cost matrix is not working correctly. I cannot quite understand looking at the cost matrix if I am penalizing the misclassification of 0 (majority class) or 1(minority class). Why first row and first column element =0 and the other is 2. Please help.

Answer 1

This can be shown using some test data, for example the following:

rng(42)
X = randn(1000, 2);
y = rand(1000, 1) >= 0.98;
X(y==1, :) = X(y==1, :) + [2, 2];

A simple SVM with Gaussian kernel function will not work well due to the class imbalance:

model = fitcsvm(X, y, 'KernelFunction', 'rbf')
confusionmat(y, model.predict(X))

ans =

   979     2
    14     5

As you have already recognized, the 'Cost' parameter can be used to compensate for the imbalance by placing a higher penalty on misclassifications of the minority class. In the two-dimensional case, the cost matrix is built up as follows:

[ Cost(0, 0),    Cost(0, 1)
  Cost(1, 0),    Cost(1, 1) ]

Now, Cost(0, 0) is the cost of classifying a sample belonging to class 0 as class 0 . This is a correct classification, so usually the cost is set to 0. Next, Cost(0, 1) is the cost of classifying a point belonging to class 0 as class 1 , ie a wrong classification.

In your example, class 0 is more likely to occur than class 1 , so we should impose a low penalty on classifying a sample from class 0 (the majority) as class 1 (the minority), and a high penalty on classifying a sample from class 1 (the minority) as class 0 (the majority). So Cost(0, 1) should be low and Cost(1, 0) should be high.

By setting c = [0, 2.2; 1, 0] c = [0, 2.2; 1, 0] , you did the opposite - you advised the fitcsvm function to rather classify a minority sample as majority than the opposite:

c = [0, 2.2; 1, 0];
model = fitcsvm(X, y, 'KernelFunction', 'rbf', 'Cost', c);
confusionmat(y, model.predict(X))

ans =

   981     0
    19     0

If you use the same weights in the cost matrix c , but switch Cost(0, 1) and Cost(1, 0) , then the desired effect will happen:

c = [0, 1; 2.2, 0];
model = fitcsvm(X, y, 'KernelFunction', 'rbf', 'Cost', c);
confusionmat(y, model.predict(X))

ans =

   973     8
     7    12

This does improve our result: over all we have a similar amount of misclassifications: 15 instead of 16 total misclassifications, but 12 out of our 19 samples of the minority class are correct with the new model, compared to only 5 before.

Answer 2

Based on your results, it seems that both classes belong to the same distribution. Try oversampling your training data (generating more positive samples using your available positive samples) and build your model on that, then test your model on the testing.