C# HMM Gesture Recognition usng Kinect

Question

I'm working on a solution to do gesture recognition using the Kinect sensor. Now I'm using Accord .NET to train the HMM. I have a dataset with saved gestures. This dataset has 11 gestures and each one has 32 frames with 18 points saved.

So I have a (double [12] [32,18]) input dataset and a (int[12]) output dataset, but when i do: double error = teacher.Run(inputSequences, output), it gives me this : "Specified argument was out of the range of valid values."

Does anyone knows how to solve this? Should treat the dataset before using it o on the hmm teacher or the dataset is ok like this?

Answer 1

I have used Accord.NET in the past and its really one of the best implementations for a HMM engine. However, when I trained my HMM, I passed the HMM parameters (namely PI, A and B) to the Baum Welch Teacher with the input data set supplied using an organized excel sheet. (similar to what Accord's author himself has used in his project). I somehow feel that since you are storing your data set as a multi-dimensional array and directly supplying it to the teacher, its unable to process it properly. Maybe you could supply one gesture record at a time or change the storage structure of your data set altogether. I advice going through the entire example of Accord if you haven't already because it worked just fine for me.

Answer 2

The problem might have been that the teaching algorithm expects the training sequences to be in the form double[12][32][18] , rather than double[12][32,18] . The training data should be a collection of sequences of multivariate points. It should also be necessary to note that, if you have 11 possible classes of gestures, the integer labels given in the int[12] array should be comprised of values between 0 and 10 only.

Thus if you have 12 gesture samples, each containing 32 frames, and each frame is a vector of 18 points, you should be feeding the teacher with a double[12][32][18] array containing the observations and a int[12] array containing the expected class labels.

The example below, extracted from the HiddenMarkovClassifierLearning documentation page should help to give an idea how the vectors should be organized!

// Create a Continuous density Hidden Markov Model Sequence Classifier 
// to detect a multivariate sequence and the same sequence backwards. 

double[][][] sequences = new double[][][]
{
    new double[][] 
    { 
        // This is the first  sequence with label = 0 
        new double[] { 0, 1 },
        new double[] { 1, 2 },
        new double[] { 2, 3 },
        new double[] { 3, 4 },
        new double[] { 4, 5 },
    }, 

    new double[][]
    {
            // This is the second sequence with label = 1 
        new double[] { 4,  3 },
        new double[] { 3,  2 },
        new double[] { 2,  1 },
        new double[] { 1,  0 },
        new double[] { 0, -1 },
    }
};

// Labels for the sequences 
int[] labels = { 0, 1 };

In the above code, we have set the problem for 2 sequences of observations, where each sequence containing 5 observations, and in which each observations is comprised of 2 values. As you can see, this is a double[2][5][2] array. The array of class labels is given by a int[2], containing only values ranging from 0 to 1.

Now, to make the example more complete, we can continue creating and training the model using the following code:

var initialDensity = new MultivariateNormalDistribution(2);

// Creates a sequence classifier containing 2 hidden Markov Models with 2 states 
// and an underlying multivariate mixture of Normal distributions as density. 
var classifier = new HiddenMarkovClassifier<MultivariateNormalDistribution>(
    classes: 2, topology: new Forward(2), initial: initialDensity);

// Configure the learning algorithms to train the sequence classifier 
var teacher = new HiddenMarkovClassifierLearning<MultivariateNormalDistribution>(
    classifier,

    // Train each model until the log-likelihood changes less than 0.0001
    modelIndex => new BaumWelchLearning<MultivariateNormalDistribution>(
        classifier.Models[modelIndex])
    {
        Tolerance = 0.0001,
        Iterations = 0,

        FittingOptions = new NormalOptions()
        {
            Diagonal = true,      // only diagonal covariance matrices
            Regularization = 1e-5 // avoid non-positive definite errors
        }
    }
);

// Train the sequence classifier using the algorithm 
double logLikelihood = teacher.Run(sequences, labels);

And now we can test the model, asserting that the output class label indeed matches what we are expecting:

// Calculate the probability that the given 
//  sequences originated from the model 
double likelihood, likelihood2;

// Try to classify the 1st sequence (output should be 0) 
int c1 = classifier.Compute(sequences[0], out likelihood);

// Try to classify the 2nd sequence (output should be 1) 
int c2 = classifier.Compute(sequences[1], out likelihood2);