Feature detection and matching - Retinal images

Question

I want to automatically register two fundus (retinal) images together using OpenCV. I have done the registration in Matlab using control points that I selected manually from each image. This approach worked very well, and I obtained really good results with affine transformations and with non-linear approaches, such as second-order and cubic polynomial transformations.

A little more about my fundus images: the two images sometimes have very different contrasts, and are rotated and translated by large values. There's nothing I can do about the translation/rotation; that's the way the data sets are.

What I have tried so far in OpenCV is I tried the various feature detector and descriptors available to extract my keypoints and find the matches. However, since my images are of lower contrast, sometimes I do not obtain matches at all (this is pretty much always the case).

I wanted to know if there are particular ideas that I should work on if I have not done so before? Any helpful pointers/hints would also go a long way. Thanks!

These were the feature detectors I tried: SIFT, SURF, MSER, ORB, and FAST

These were the feature descriptors I tried: FREAK, SURF, BRIEF, ORB, and FAST

Examples of my output are shown below:

EDIT: Links to original images: Image1 Image 2

EDIT: This image shows the two input images, side-by-side.

EDIT: This image shows the matches between the two images after applying the ORB feature detector, and the FREAK feature descriptor. The matches are bad. (obviously)

Answer 1

• You should definitely include SIFT in the descriptors. It's the go-to descriptor.

• With any of the detectors and descriptors you will have reasonably large numbers of outliers in matching. Therefore it is important to estimate any transformation in a robust way. Here, the solution that works well in most cases is RANSAC. There is a RANSAC implementation available in OpenCV.

Update

Your images are quite different to normal (photographic) images.

Even for me it is difficult to say how the image pairs are transformed. When I do see it, I look at longer lines and their branches, not at small patches. So, I wouldn't expect a algorithm to do it.

---

I'd experiment with the parameters available in the detectors.

For SIFT those are described and can be found here (Matlab vl_feat). Particularly, I'd look at edge_threshold and peak_threshold , and maybe oversample the image by using the -1th octave .

In OpenCV the documentation provides the following parameters, all of which you want to change so as to get more detections.

• nfeatures – The number of best features to retain. The features are ranked by their scores (measured in SIFT algorithm as the local contrast)
• OctaveLayers – The number of layers in each octave. 3 is the value used in D. Lowe paper. The number of octaves is computed automatically from the image resolution.
• contrastThreshold – The contrast threshold used to filter out weak features in semi-uniform (low-contrast) regions. The larger the threshold, the less features are produced by the detector.
• edgeThreshold – The threshold used to filter out edge-like features. Note that the its meaning is different from the contrastThreshold, ie the larger the edgeThreshold, the less features are filtered out (more features are retained).
• sigma – The sigma of the Gaussian applied to the input image at the octave #0. If your image is captured with a weak camera with soft lenses, you might want to reduce the number.

Another Thought

There is a paper by Efros' group where they describe a descriptor that might be suitable to your problem. The descriptor is geometric in nature. They take one interest point and then use the four closest neighbours around it to compute the descriptor. These four points form two triangles, the ratio of the areas of the triangles becomes the descriptor. I'll have to find the paper for details.