Background Substraction with user input

Question

I am looking for an algorithm or, even better, some library that covers background substraction from a single static image (no background model available). What whould be possible though is some kind of user input like for example https://clippingmagic.com does it. Sadly my google fu is bad here as i cant find any papers on that topic with my limited amount of keywords.

Answer 1

That webpage is really impressive. If I were to try and implement something similar I would probably use k-means clustering using the CIELAB colorspace . The reason for changing the colorspace is so that colors can be represented by two points rather than 3 as a regular RGB image. This should speed up clustering. Additionally, the CIELAB color space was build for this purpose, finding "distances" (similarities) between colors and accounts for the way humans perceive color. Not just looking at the raw binary data the computer has.

But a quick overview of kmeans. For this example we will say k=2 (meaning only two clusters)

1. Initialize each cluster with a mean.
2. Go through every pixel in your image and decide which mean it is closer to, cluster 1 or 2?
3. Compute the new mean for your clusters after you've processed all the pixels
4. using the newly computed means repeat steps 2-4 until convergence (meaning the means don't change very much)

Now that would work well when the foreground image is notably different than the background. Say a red ball in a blue background, but if the colors are similar it would be more problematic. I would still stick to kmeans but have a larger number of clusters. So on that web page you can make multiple red or green selections. I would make each of these strokes a cluster, and intialize my cluster to the mean. So say I drew 3 red strokes, and 2 green ones. That means I'd have 5 groups. But somehow internally I add an extra attribute as foreground/background. So that each cluster will have a small variance, but in the end, I would only display that attribute, foreground or background. I hope that made sense.

Maybe now you have some search terms to start off with. There may be many other methods but this is the first I thought of, good luck.

EDIT

After playing with the website a bit more I see it uses spatial proximity to cluster. So say I had 2 identical red blobs on opposite sides of the image. If I only annotate the left side of the image the blob on the right side might not get detected. Kmeans wouldn't replicate this behavior since the method I described only uses the color to cluster pixels completely oblivious to their location in the image.

I don't know what tools you have at your disposal but here is a nice matlab example/tutorial on color based kmeans