如何使用两个特征向量训练svm？

Question

I have two numpy arrays(features). The dimensions of numpy arrays are :

audio=(360,13) ---> Features are extracted from audio files

image=(360,5)--> Features are extracted from the spectrogram of these audio files.

I want to use these two arrays together to train svm classifier. But I know svm train get just one array. (svm.train(feature , label)). I am looking for is there anything like svm.train(audio,image,label)

I also tried to concatenate these two arrays but the dimensions are different. How can I solve this situation ?

Answer 1

While @Saedeas provided a simple solution, I would suggest going in slightly different way.

Concatenation is good for homogenous features, it does not work well when data comes from completely different modalities (like audio + video). However, one can deal with that using simple properties of kernel functions (which are the base of SVMs), namely sum of two kernels is a kernel, so we can define:

K_{audio x video}(x,y) = a K_{video}(x_{video}, y_{video}) + 
                         (1-a) K_{audio}(x_{audio},y_{audio})

so given kernels for each modality separately, we define a joint kernel on top of it, where a is a hyperparameter ae [0,1] to be adjusted.

Code-wise it might be done in a similar way to what is already suggested:

# First concat, but only for easier handling
new_data = np.concatenate((audio,image), axis=1)
y = ...

def video_kernel(X, Y):
  ...

def audio_kernel(X, Y):
  ...

# now new kernel
def new_kernel(X, Y, a=0.5):
  return a*audio_kernel(X[:, :13], Y[:, :13]) + (1-a)*video_kernel(X[:, 13:], Y[:, 13:])

svm = SVC(kernel=new_kernel)
svm.fit(new_data, y)

Answer 2

Maybe I'm misreading, but how are the dimensions an issue?

You have 360 samples in each array, one has 13 dimensions, the other has 5. Turn this into a single array of 360 samples with 18 dimensions.

You may need to normalize the values, but the concatenation should just be:

new_data = np.concatenate((audio,image), axis=1)

You might normalize the data values by making them zero mean, unit variance (find the mean and variance in each dimension, subtract the means from every sample, and divide by the variances).

means = np.mean(new_data, axis=0)
vars = np.var(new_data, axis=0)
norm_data = (new_data - means) / vars

Edit: You might still normalize in this way, but I'd use @lejlot's solution. The multiple kernel approach makes a lot of sense and is more flexible than this approach.

Answer 3

You can concatenate them. For example like there: http://scikit-learn.org/stable/modules/pipeline.html#feature-union , or you can train two SVM and work with prediction results. Example: http://scikit-learn.org/stable/modules/ensemble.html#voting-classifier .