Creating fisher vectors using OpenIMAJ

Question

I'm trying to classify images using fisher vectors as described in: Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013). Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision, 105(3), 222–245. http://doi.org/10.1007/s11263-013-0636-x

To try out and evaluate this method I want to use the OpenIMAJ library, since according to their JavaDoc they use exactly this method to create the fisher vectors, but I can't get it work. I've tried creating the SIFT feature vectors with OpenIMAJ and OpenCV but for both I get the same error: EM algorithm was never able to compute a valid likelihood given initial parameters. Try different init parameters (or increasing n_init) or check for degenerate data.

If someone has already experience using this method I would greatly appreciate any help. I've created a little example which should illustrate the problem:

    // load an image
    LocalFeatureList<Keypoint> findFeatures = new DoGSIFTEngine()
            .findFeatures(ImageUtilities
                    .readMBF(
                            new URL(
                                    "http://upload.wikimedia.org/wikipedia/en/2/24/Lenna.png"))
                    .flatten());

    // convert to double array
    double[][] data = new double[findFeatures.size()][findFeatures.get(0)
            .getDimensions()];
    for (int i = 0; i < findFeatures.size(); i++) {
        data[i] = findFeatures.get(i).getFeatureVector().asDoubleVector();
    }

    GaussianMixtureModelEM gaussianMixtureModelEM = new GaussianMixtureModelEM(
            64, CovarianceType.Diagonal);

    // error is thrown here
    MixtureOfGaussians estimate = gaussianMixtureModelEM.estimate(data);

Answer 1

I would guess the problem is that you're not using enough data. To train the GMM you should be using many samples from across the training corpus, rather than a single image. Note also, that you should also be applying PCA to reduce the dimensionality of the features before learning the GMM (this isn't strictly required, but really helps performance as shown in the paper you linked).

Once this is done, you can then use the OpenIMAJ FisherVector class to actually compute the vector from the SIFT points for each image.

As an aside - as you're doing classification, you almost certainly want to be using a DenseSIFT variant over DoG-SIFT if you want any kind of decent performance.

Below is sample code for building FisherVectors from the first 100 images of the UKBench dataset:

    //Load features from disk
    final List<MemoryLocalFeatureList<FloatKeypoint>> data = new ArrayList<MemoryLocalFeatureList<FloatKeypoint>>();
    final List<FloatKeypoint> allKeys = new ArrayList<FloatKeypoint>();

    for (int i = 0; i < 100; i++) {
        final MemoryLocalFeatureList<FloatKeypoint> tmp = FloatKeypoint.convert(MemoryLocalFeatureList.read(
                new File(String.format("/Users/jsh2/Data/ukbench/sift/ukbench%05d.jpg", i)), Keypoint.class));
        data.add(tmp);
        allKeys.addAll(tmp);
    }

    //randomise their order
    Collections.shuffle(allKeys);

    //sample 1000 of them to learn the PCA basis with 64 dims
    final double[][] sample128 = new double[1000][];
    for (int i = 0; i < sample128.length; i++) {
        sample128[i] = ArrayUtils.convertToDouble(allKeys.get(i).vector);
    }

    System.out.println("Performing PCA " + sample128.length);
    final ThinSvdPrincipalComponentAnalysis pca = new ThinSvdPrincipalComponentAnalysis(64);
    pca.learnBasis(sample128);

    //project the 1000 training features by the basis (for computing the GMM)
    final double[][] sample64 = pca.project(new Matrix(sample128)).getArray();

    //project all the features by the basis, reducing their dimensionality
    System.out.println("Projecting features");
    for (final MemoryLocalFeatureList<FloatKeypoint> kpl : data) {
        for (final FloatKeypoint kp : kpl) {
            kp.vector = ArrayUtils.convertToFloat(pca.project(ArrayUtils.convertToDouble(kp.vector)));
        }
    }

    //Learn the GMM with 128 components
    System.out.println("Learning GMM " + sample64.length);
    final GaussianMixtureModelEM gmmem = new GaussianMixtureModelEM(128, CovarianceType.Diagonal);
    final MixtureOfGaussians gmm = gmmem.estimate(sample64);

    //build the fisher vector representations
    final FisherVector<float[]> fisher = new FisherVector<float[]>(gmm, true, true);

    int i = 0;
    final double[][] fvs = new double[100][];
    for (final MemoryLocalFeatureList<FloatKeypoint> kpl : data) {
        fvs[i++] = fisher.aggregate(kpl).asDoubleVector();
    }