What is speckle in stereo BM and SGBM algorithm implemented in OpenCV

Question

While applying the stereo BM & SGBM algorithms implemented in OpenCV, I came across the notion of "Speckle noise", that are filtered by a speckle filter, caracterized by its "speckleWindowSize" & "speckleRange parameters" => see openCV's documentation on that link to OpenCV doc

First of all, what is Speckle noise and what causes it ?

Secondly, in the above link, you can find the below definitions (which don't really explain anything and just give ranges that seem to come from nowhere) :

"speckleWindowSize : Maximum size of smooth disparity regions to consider their noise speckles and invalidate. Set it to 0 to disable speckle filtering. Otherwise, set it somewhere in the 50-200 range."

"speckleRange : Maximum disparity variation within each connected component. If you do speckle filtering, set the parameter to a positive value, it will be implicitly multiplied by 16. Normally, 1 or 2 is good enough. "

while the famous "Learning OpenCV" book from Gary Bradski and Adrian Kaehler give a totally other range for speckleWindowSize :

"block-based matching has problems near the boundaries of objects because the matching window catches the foreground on one side and the background on the other side. Th is results in a local region of large and small disparities that we call speckle. To prevent these borderline matches, we can set a speckle detector over a speckle window (ranging in size from 5-by-5 up to 21 by-21) by setting speckleWindowSize, which has a default setting of 9 for a 9-by-9 window. Within the speckle window, as long as the minimum and maximum detected disparities are within speckleRange, the match is allowed (the default range is set to 4)."

By testing it, it seems that I can effectively go up to a 200 window size, but if that is represented in pixels, isn't that a huge window ?

Also, the above text gives an explanation on what speckle is. As I understand it, we juste have small values of disparities for the background and large values for the foreground, which is exactly what is supposed to be...? Therefore, I don't understand why it is considered as noise and why we should filter it ?

Any help would be appreciated,

Thank you.

Answer 1

While using any of provided disparity algorithms it's likely to have better results if post filtering is applied. Typical problem zones of disparity maps from stereo images are object edges , shaded areas , textured regions comes from how disparity map is counted. You may check this tutorial where one type of post filtering is applied to BM disparity algorithm.

"Learning OpenCV" is a great book and your cite from it gives a clear answer to your question.

The is results in a local region of large and small disparities that we call speckle.

I took an image from the question at answers.opencv.org .

Speckle is a region with huge variance between counted disparities which should be considered as a noise (and filtered) . And speckles are likely to come in problem areas.

The reason for manual setup of speckle-related parameters of algorithm is that this parameters will very between different scenes and setups. So there is not a single optimal choice of speckleWindowSize and speckleRange to fit any developer's requirements. You may work with large objects close to camera (like on the image) or with small objects far from camera and close to background (cars on bird-view road scene) etc. So you should set parameters which suit your particular camera setup (or provide your user with interface to adjust them if camera setups may vary). Consider areas around fingers and inside a palm. There are speckles (especially area inside a palm). The difference in disparity is noise in this case and should be filtered. Choosing very big speckleWindowSize (blue rectangle) will lead to loss of small but important details like fingers. It maybe better to choose smaller speckleWindowSize (red rectangle) and bigger speckleRange since disparity variation seems to be big.