How to compute accuracy, F1-score, recall, precision and EER for one-class SVM in python?

Question

I have a touch dataset collected by a user, this dataset is 1000x31 matrix of floating numbers. How can I apply one-class SVM classifier to detect an anomaly detection? How I can implement this in python and compute accuracy, recall, precision, F1-score, and EER?

I started with this link but I couldn't compute the above performance metrics!! https://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py

Answer 1

First, structure your data by finding out which of these 1000 examples are from the class that will be trained by OCSVM. Then separate into train and test for the OCSVM to use the train set for training and the test set for validating (generate metrics like accuracy, recall, F1, accuracy).

If all 1000 examples are of the class of interest, you will only be able to find the recall (TP/(TP+FN)) since you only have positive documents that can be classified as true positives (TP, ie, OCSVM got it right) or false negatives (FN, i.ie, the OCSVM was wrong).

To generate the precision (TP/(TP+FP)), F1, and accuracy (TP+TN/(TP+TN+FP+FN)), you need to create some examples that do not belong to the class of interest since with these contrary examples, OCSVM can generate true negatives (TN) and false positives (FP).

Here is a code that returns the metrics given a set of documents of the class of interest for train and test, and a set of the non-class of interest for testing.

from sklearn.model_selection import train_test_split
from sklearn.svm import OneClassSVM
from sklearn.metrics import classification_report

def evaluation_one_class(preds_interest, preds_outliers):
  y_true = [1]*len(preds_interest) + [-1]*len(preds_outliers)
  y_pred = list(preds_interest)+list(preds_outliers)
  return classification_report(y_true, y_pred, output_dict=False)

def evaluate_model(X_train, X_test, X_outlier, model):
  
  one_class_classifier = model.fit(X_train)

  Y_pred_interest = one_class_classifier.predict(X_test)
  
  Y_pred_ruido = one_class_classifier.predict(X_outlier)

  print(evaluation_one_class(Y_pred_interest, Y_pred_ruido))


class_of_interest = ''

df_interest = df[df['target'] == class_of_interest]
df_outlier = df[df['target'] != class_of_interest]

df_train_int, df_test_int = train_test_split(df_interest,test_size=0.30, random_state=25) 

clf = OneClassSVM(gamma='auto')

evaluate_model(df_train_int, df_test_int, df_outlier, clf)