Implementation of SVM for classification without library in c++

Question

I'm studying Support Vector Machine last few weeks. I understand the theoretical concept how I can classify a data into two classes. But its unclear to me how to select support vector and generate separating line to classify new data using C++.

Suppose, I have two training data set for two classes

After plotting data, I get the following feature space with vector and here, separating line is also clear.

How to implement this in C++ without library functions. It will help me to clear my implementation concept about SVM. I need to be clear about implementation as I'm going to apply SVM in opinion mining for my native language.

Answer 1

I will join to most people's advice and say that you should really consider using a library. SVM algorithm is tricky enough to add the noise if something is not working because of a bug in your implementation. Not even talking about how hard is to make an scalable implementation in both memory size and time.

That said and if you want to explore this just as a learning experience, then SMO is probably your best bet. Here are some resources you could use:

The Simplified SMO Algorithm - Stanford material PDF

Fast Training of Support Vector Machines - PDF

The implementation of Support Vector Machines using the sequential minimal optimization algorithm - PDF

Probably the most practical explanation that I have found is the one on the chapter 6 of the book Machine Learning in action by Peter Harrington. The code itself is on Python but you should be able to port it to C++. I don't think it is the best implementation but it might be good enough to have an idea of what is going on.

The code is freely available:

https://github.com/pbharrin/machinelearninginaction/tree/master/Ch06

Unfortunately there is not sample for that chapter but a lot of local libraries tend to have this book available.

Answer 2

Most cases SVM is trained with SMO algorithm -- a variation of coordinate descent that especially suits the Lagrangian of the problem. It is a bit complicated, but if a simplified version will be ok for your purposes, I can provide a Python implementation. Probably, You will be able to translate it to C++

class SVM:
  def __init__(self, kernel='linear', C=10000.0, max_iter=100000, degree=3, gamma=1):
    self.kernel = {'poly'  : lambda x,y: np.dot(x, y.T)**degree,
                   'rbf'   : lambda x,y: np.exp(-gamma*np.sum((y - x[:,np.newaxis])**2, axis=-1)),
                   'linear': lambda x,y: np.dot(x, y.T)}[kernel]
    self.C = C
    self.max_iter = max_iter

  def restrict_to_square(self, t, v0, u):
    t = (np.clip(v0 + t*u, 0, self.C) - v0)[1]/u[1]
    return (np.clip(v0 + t*u, 0, self.C) - v0)[0]/u[0]

  def fit(self, X, y):
    self.X = X.copy()
    self.y = y * 2 - 1
    self.lambdas = np.zeros_like(self.y, dtype=float)
    self.K = self.kernel(self.X, self.X) * self.y[:,np.newaxis] * self.y
    
    for _ in range(self.max_iter):
      for idxM in range(len(self.lambdas)):
        idxL = np.random.randint(0, len(self.lambdas))
        Q = self.K[[[idxM, idxM], [idxL, idxL]], [[idxM, idxL], [idxM, idxL]]]
        v0 = self.lambdas[[idxM, idxL]]
        k0 = 1 - np.sum(self.lambdas * self.K[[idxM, idxL]], axis=1)
        u = np.array([-self.y[idxL], self.y[idxM]])
        t_max = np.dot(k0, u) / (np.dot(np.dot(Q, u), u) + 1E-15)
        self.lambdas[[idxM, idxL]] = v0 + u * self.restrict_to_square(t_max, v0, u)
    
    idx, = np.nonzero(self.lambdas > 1E-15)
    self.b = np.sum((1.0 - np.sum(self.K[idx] * self.lambdas, axis=1)) * self.y[idx]) / len(idx)
  
  def decision_function(self, X):
    return np.sum(self.kernel(X, self.X) * self.y * self.lambdas, axis=1) + self.b

In simple cases it works not much worth than sklearn.svm.SVC, comparison shown below

For more elaborate explanation with formulas you may want to refer to this ResearchGate preprint . Code for generating images can be found on GitHub .