Saving ORB feature vectors using OpenCV4Android (java API)

Question

I have a training set of images, for each of which I've detected and computed their feature vectors (using ORB feature descriptors and extractors. The questions is : since I need to save those features to reutilise them for matching against test images (using SVM classifier); what is the best way to store the feature vectors, locally on the Android device?

The feature vectors to be saved are of variable size per image, and are thus those with non-maximal sizes are padded with zeros to unify all vectors' sizes. The maximum size currently is 500 rows x 32 cols; thus 16k features.

here are the options I could reach so far;

• I've heard of OpenCV's FileStorage , but when going through the java documentation, I noticed a save method for HOG features (not ORB). Furthermore, I'm not sure if saving features using OpenCV's file storage options would be most optimal memory-wise for Android phones, given that the xml file would be too large to load.
• My current choice is to opt for a sqlLite database, having a table with two cols; id and feature (as frequently suggested online); to tabulate all the 16k features in sqlLite. That seems rather phone-storage intensive, but it's the most reasonable solution I can find.

Is there a common method to handling feature vectors on Android phones? Does it encompass any of the above methods; if not can you please offer some guidelines on how to implement such a storage solution?

Thank you.

Answer 1

In my opinion the most universal way to store the keypoints is to first convert them to a data-interchange format like JSON.

After you are able to do that conversion you have a lot of flexibility to store it. JSON is easily converted to a String and/or sent through a network connection.

With OpenCV C++ you are able to store data as YAML , but that is not available for Android yet.

To parse JSON in Java you can use this easy to use library Google GSON .

And here is my first attempt to do exactly that:

 public static String keypointsToJson(MatOfKeyPoint mat){
    if(mat!=null && !mat.empty()){          
        Gson gson = new Gson();

        JsonArray jsonArr = new JsonArray();            

        KeyPoint[] array = mat.toArray();
        for(int i=0; i<array.length; i++){
            KeyPoint kp = array[i];

            JsonObject obj = new JsonObject();

            obj.addProperty("class_id", kp.class_id); 
            obj.addProperty("x",        kp.pt.x);
            obj.addProperty("y",        kp.pt.y);
            obj.addProperty("size",     kp.size);
            obj.addProperty("angle",    kp.angle);                          
            obj.addProperty("octave",   kp.octave);
            obj.addProperty("response", kp.response);

            jsonArr.add(obj);               
        }

        String json = gson.toJson(jsonArr);         

        return json;
    }
    return "{}";
}

public static MatOfKeyPoint keypointsFromJson(String json){
    MatOfKeyPoint result = new MatOfKeyPoint();

    JsonParser parser = new JsonParser();
    JsonArray jsonArr = parser.parse(json).getAsJsonArray();        

    int size = jsonArr.size();

    KeyPoint[] kpArray = new KeyPoint[size];

    for(int i=0; i<size; i++){
        KeyPoint kp = new KeyPoint(); 

        JsonObject obj = (JsonObject) jsonArr.get(i);

        Point point = new Point( 
                obj.get("x").getAsDouble(), 
                obj.get("y").getAsDouble() 
        );          

        kp.pt       = point;
        kp.class_id = obj.get("class_id").getAsInt();
        kp.size     =     obj.get("size").getAsFloat();
        kp.angle    =    obj.get("angle").getAsFloat();
        kp.octave   =   obj.get("octave").getAsInt();
        kp.response = obj.get("response").getAsFloat();

        kpArray[i] = kp;
    }

    result.fromArray(kpArray);

    return result;
}

Answer 2

I would suggest storing the feature vectors as images to have a simple and compact representation. You could even use non-destructive compression such as png to minimize file size.

Answer 3

I see that you have considered using the Android SQLite database:

My current choice is to opt for a sqlLite database, having a table with two cols; id and feature (as frequently suggested online); to tabulate all the 16k features in sqlLite. That seems rather phone-storage intensive, but it's the most reasonable solution I can find.

There is a way to save and retrieve MatOfKeyPoint to SQLite database with reasonable efficiency.

Using the database has the advantage of not needing to request write external storage permission from the user (although that permission might be needed for some other of your apps functions).

There is a complete Android solution, with Java code which can be found in this StackOverflow Answer .

The following is a description of what's going on in the code from that answer...

MatOfKeyPoint to byte[] and some attributes

To save to the database, you need to save to a byte[] object. Using the MatOfKeyPoint.get() method, you can get a populated float[] . Then using ByteBuffer.putFloat() , you can loop through all of your floats, finally getting a populated byte[] by using ByteBuffer.array() .

You also need to save some attirbutes MatOfKeyPoint.rows() , MatOfKeyPoint.cols() , and MatOfKeyPoint.type() to your database, along with the blob of byte[] .

Database blob (byte[]) to MatOfKeyPoint

To reconstitute your MatOfKeyPoint object from the database, first you make a float[] out of your blob. Use ByteBuffer.wrap(blob) , then run ByteBuffer.asFloatBuffer() , and finally FloatBuffer.get() with a properly sized new float[] .

Now that you have yourFloatArray you run MatOfKeyPoint.create(rows, cols, type) , those three things coming from the database record. Finally you run MatOfKeyPoint.put(0, 0, yourFloatArray) .