Testing my SVM Model with different parameters yields exact same results

Question

My goal is to use an SVM w/ HOG features to classify vehicles in traffic under sedans and SUVs.

I've used various kernels (RBF, LINEAR, POLY) and each give different results, but they give the same results no matter the parameters changed. For example, if I am using a POLY kernel and the degree is greater than or equal to .65 it will classify everything as an SUV, if its less than .65 then it will classify all my testing images as sedans.

With a LINEAR kernel, the only parameter changed is C. No matter what the parameter C is, I always get 8/10 images classified as sedans and the same 2 classified as SUVs.

Now I only have about 70 training images and 10 testing images, I haven't been able to find a good dataset of vehicles from the rear and up like from a bridge that I will be using this for. Could the problem be due to this small dataset, or the parameters, or something else? Also, I see that my support vectors are usually very high, like 58 out of the 70 training images, so that may be a problem with the dataset? Is there a way for me to visualize the training points somehow--in the SVM examples they always have a nice 2D plot of points and draw a line through it, but is there a way to plot those points with images so I can see if my data is linearly separable and make adjustments accordingly? Are my HOG parameters accurate for a 150x200 image of a car?

Also note that when I use testing images that are the same as training images, then the SVM model predicts perfectly, but obviously that's cheating.

The following image shows the result, and an example of a testing image

Here is my code, I didn't include most of it because I'm not sure the code is the problem. First I take the positive images, extract HOG features, then load them into the training Mat, and then do the same for the negative images in the same way that I do for the included testing part.

    //Set SVM Parameters (not sure about these values, but just wanna see something)
Ptr<SVM> svm = SVM::create();
svm->setType(SVM::C_SVC);
svm->setKernel(SVM::POLY);
svm->setC(50);
svm->setGamma(100);
svm->setDegree(.65);
//svm->setTermCriteria(TermCriteria(TermCriteria::MAX_ITER, 100, 1e-6));

cout << "Parameters Set..." << endl;

svm->train(HOGFeat_train, ROW_SAMPLE, labels_mat);

Mat SV = svm->getSupportVectors();
Mat USV = svm->getUncompressedSupportVectors();

cout << "Support Vectors: " << SV.rows << endl;
cout << "Uncompressed Support Vectors: " << USV.rows << endl;

cout << "Training Successful" << endl;

waitKey(0);

//TESTING PORTION

cout << "Begin Testing..." << endl;

int num_test_images = 10;
Mat HOGFeat_test(1, derSize, CV_32FC1); //Creates a 1 x descriptorSize Mat to house the HoG features from the test image

for (int file_count = 1; file_count < (num_test_images + 1); file_count++)
{

    test << nameTest << file_count << type;     //'Test_1.jpg' ... 'Test_2.jpg' ... etc ...
    string filenameTest = test.str();
    test.str("");

    Mat test_image = imread(filenameTest, 0);           //Read the file folder

    HOGDescriptor hog_test;// (Size(64, 64), Size(32, 32), Size(16, 16), Size(32, 32), 9, 1, -1, 0, .2, 1, 64, false);
    vector<float> descriptors_test;
    vector<Point> locations_test;

    hog_test.compute(test_image, descriptors_test, Size(64, 64), Size(0, 0), locations_test);

    for (int i = 0; i < descriptors_test.size(); i++)
        HOGFeat_test.at<float>(0, i) = descriptors_test.at(i);

    namedWindow("Test Image", CV_WINDOW_NORMAL);
    imshow("Test Image", test_image);

    //Should return a 1 if its an SUV, or a -1 if its a sedan
    float result = svm->predict(HOGFeat_test);

    if (result <= 0)
        cout << "Sedan" << endl;
    else
        cout << "SUV" << endl;

    cout << "Result: " << result << endl;

    waitKey(0);
}

Answer 1

Two things solved this issue:

1) I got a larger dataset of vehicles. I used about 400 SUV images and 400 sedan images for the training portion and then another 50 images for the testing portion.

2) In: Mat HOGFeat_test(1, derSize, CV_32FC1), I had the wrong derSize by about an order of magnitude larger. The actual size was 15120, but I had the Mat have 113400 columns. Thus, I filled only about 10% of the testing mat with useful feature data, so it was much harder for the SVM to tell any difference between SUVs and Sedans.

Now it works great with both the linear and poly kernel (C = 10), and my accuracy is better than I expected at a whopping 96%.