Prediction always the same while using Sci-kit Learn SVM

Question

I have a dataset where I'm trying to predict what kind of DNA a data entry is from the DNA makeup. For example, the string ATTAG...ACGAT might translate to EI . The possible outputs are either EI , IE , or N . The dataset can be investigated further here . I tried switching out kernels from linear to rbf but the results are the same. The SVM classifier seems to output N everytime. Any ideas why? I'm a beginner to Sci-kit Learn.

import pandas as pd
# 3190 total
training_data = pd.read_csv('new_training.csv')
test_data = pd.read_csv('new_test.csv')
frames = [training_data, test_data]
data = pd.concat(frames)
x = data.iloc[:, 0:59]
y = data.iloc[:, 60]

x = pd.get_dummies(x)
train_x = x.iloc[0:3000, :]
train_y = y.iloc[0:3000]
test_x = x.iloc[3000:3190, :]
test_y = y.iloc[3000:3190]

from sklearn import svm
from sklearn import preprocessing

clf = svm.SVC(kernel="rbf")
label_encoder = preprocessing.LabelEncoder()
label_encoder.fit(y)

print(label_encoder.transform(train_y))
clf.fit(train_x, label_encoder.transform(train_y))

for u in train_y.unique():
    print(u)

predictions = clf.predict(test_x)

correct = 0
total = len(predictions)
for i in range(total):
    prediction = label_encoder.inverse_transform(predictions[i])
    print('predicted %s and actual %s' % (prediction, test_y[i]))
    print(len(prediction))
    if prediction == test_y[i]:
        correct += 1

print('correct %d out of %d' % (correct, total))

First I import the training and test data, combine it and split it into as either x (inputs) or y (output label). Then I convert x into the dummy variable version from the original 60 columns to like 300~ columns since each DNA spot can be A , T , G , C and sometimes N . Basically have either a 0 or 1 for all the possible inputs for each input. (Is there a better way to do this? Sci-kit learn doesn't support categorical encoding and I tried best I could from this .) Then I separate the data again (I had to merge so that I can generate dummies on the whole data space).

From here, I just run the svm stuff to fit the x and y labels and then to predict on test_x . I also had to encode/label y , from the string version to the numerical version. But yeah, it always produced N which I feel like is wrong. How do I fix? Thank you!

Answer 1

I think the issue is the way data is splitted into train and test. You have taken the first 3000 samples for training and the remaining 190 samples for testing. I found out that with such training the classifier yields the true class label for all the test samples ( score = 1.0). I have also noticed that the last 190 samples of the dataset have the same class label , namely 'N' . Therefore the result you obtained is correct.

I would recommend you to split the dataset into train and test through ShuffleSplit with test_size=.06 (this corresponds approximately to 190/3190 although to make visualization of results easier I used test_size=.01 in the sample run below). For the sake of simplicity I would also suggest you to use OneHotEncoder to encode the categorical values of the features.

Here's the full code (I have taken the liberty to perform some refactoring):

import numpy as np
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from sklearn.model_selection import ShuffleSplit
from sklearn import svm

data = np.loadtxt(r'splice.data', delimiter=',', dtype='string')

bases = {'A': 0, 'C': 1, 'D': 2, 'G': 3, 'N': 4, 'R': 5, 'S': 6, 'T': 7}

X_base = np.asarray([[bases[c] for c in seq.strip()] for seq in data[:, 2]])
y_class = data[:, 0]

enc = OneHotEncoder(n_values=len(bases))
lb = LabelEncoder()

enc.fit(X_base)  
lb.fit(y_class)

X = enc.transform(X_base).toarray()
y = lb.transform(y_class)

rs = ShuffleSplit(n_splits=1, test_size=.01, random_state=0)
train_index, test_index = rs.split(X).next()
train_X, train_y = X[train_index], y[train_index]
test_X, test_y = X[test_index], y[test_index]

clf = svm.SVC(kernel="rbf")
clf.fit(train_X, train_y)

predictions = clf.predict(test_X)

Demo:

Out[2]: 
array(['IE', 'EI', 'EI', 'EI', 'EI', 'IE', 'N', 'N', 'EI', 'N', 'N', 'IE',
       'IE', 'N', 'N', 'IE', 'EI', 'N', 'N', 'EI', 'IE', 'EI', 'IE', 'N',
       'EI', 'N', 'IE', 'N', 'EI', 'N', 'N', 'EI'], 
      dtype='|S79')

In [3]: y_class[test_index]
Out[3]: 
array(['IE', 'EI', 'EI', 'EI', 'EI', 'IE', 'N', 'N', 'EI', 'N', 'N', 'IE',
       'IE', 'N', 'N', 'IE', 'EI', 'N', 'N', 'EI', 'IE', 'EI', 'IE', 'N',
       'IE', 'N', 'IE', 'N', 'EI', 'N', 'N', 'EI'], 
      dtype='|S79')

In [4]: clf.score(test_X, test_y)
Out[4]: 0.96875

Note : Make certain your sklearn version is 0.18.1, otherwise the code above might not work.