如何从CIAreaHistogram中提取主色？

Question

I am looking to analyze the most dominant color in a UIImage on iOS (color present in the most pixels) and I stumbled upon Core Image's filter based API, particularly CIAreaHistogram.

It seems like this filter could probably help me but I am struggling to understand the API. Firstly it says the output of the filter is a one-dimensional image which is the length of your input-bins and one pixel in height. How do I read this data? I basically want to figure out the color-value with the highest frequency so I am expecting the data to contain some kind of frequency count for each color, its not clear to me how this one-dimensional image would represent that because it does not really explain the data I can expect inside this 1-d image. And if its truly a histogram why would it not return a data-structure representing that like a dictionary

Second, in the API it asks for a number of bins? What should that input be? If I want an exact analysis would the input bin parameter be the color-space of my image? What does making the bin value smaller do, I would imagine it just approximates nearby colors via Euclidean distance to the nearest bin. If this is the case will that not yield exact histogram results, why would anyone want to do that?

Any input on the above two questions from an API perspective would help me greatly

Answer 1

Ian Ollmann's idea of calculating the histogram just for the hue is really neat and can be done with a simple color kernel. This kernel returns a monochrome image of just the hue of an image (based on this original work )

let shaderString = "kernel vec4 kernelFunc(__sample c)" +
"{" +
"    vec4 K = vec4(0.0, -1.0 / 3.0, 2.0 / 3.0, -1.0);" +
"    vec4 p = mix(vec4(c.bg, K.wz), vec4(c.gb, K.xy), step(c.b, c.g));" +
"    vec4 q = mix(vec4(p.xyw, c.r), vec4(c.r, p.yzx), step(p.x, c.r));" +

"    float d = q.x - min(q.w, q.y);" +
"    float e = 1.0e-10;" +
"    vec3 hsv = vec3(abs(q.z + (q.w - q.y) / (6.0 * d + e)), d / (q.x + e), q.x);" +
"    return vec4(vec3(hsv.r), 1.0);" +
"}"

let colorKernel = CIColorKernel(string: shaderString)

If I get the hue of an image of a blue sky, the resulting histogram looks like this:

...while a warm sunset gives a histogram like this:

So, that looks like a good technique to get the dominant hue of an image.

Simon

Answer 2

CIAreaHistogram returns an image where the reg, green, blue and alpha values of each of the pixels indicates the frequency of that tone in the image. You can render that image to an array of UInt8 to look at the histogram data. There's also an undocumented outputData value:

let filter = CIFilter(
    name: "CIAreaHistogram",
    withInputParameters: [kCIInputImageKey: image])!
let histogramData = filter.valueForKey("outputData")

However, I've found vImage to be a better framework for working with histograms. First off, you need to create a vImage image format:

var format = vImage_CGImageFormat(
    bitsPerComponent: 8,
    bitsPerPixel: 32,
    colorSpace: nil,
    bitmapInfo: CGBitmapInfo(
        rawValue: CGImageAlphaInfo.PremultipliedLast.rawValue),
    version: 0,
    decode: nil,
    renderingIntent: .RenderingIntentDefault)

vImage works with image buffers that can be created from CGImage rather than CIImage instances (you can create one with the createCGImage method of CIContext . vImageBuffer_InitWithCGImage will create an image buffer:

var inBuffer: vImage_Buffer = vImage_Buffer()

vImageBuffer_InitWithCGImage(
    &inBuffer,
    &format,
    nil,
    imageRef,
    UInt32(kvImageNoFlags))

Now to create arrays of Uint which will hold the histogram values for the four channels:

let red = [UInt](count: 256, repeatedValue: 0)
let green = [UInt](count: 256, repeatedValue: 0)
let blue = [UInt](count: 256, repeatedValue: 0)
let alpha = [UInt](count: 256, repeatedValue: 0)

let redPtr = UnsafeMutablePointer<vImagePixelCount>(red)
let greenPtr = UnsafeMutablePointer<vImagePixelCount>(green)
let bluePtr = UnsafeMutablePointer<vImagePixelCount>(blue)
let alphaPtr = UnsafeMutablePointer<vImagePixelCount>(alpha)

let rgba = [redPtr, greenPtr, bluePtr, alphaPtr]

let histogram = UnsafeMutablePointer<UnsafeMutablePointer<vImagePixelCount>>(rgba)

The final step is to perform the calculation, which will populate the four arrays, and free the buffer's data:

vImageHistogramCalculation_ARGB8888(&inBuffer, histogram, UInt32(kvImageNoFlags))

free(inBuffer.data)

A quick check of the alpha array of an opaque image should yield 255 zeros with the final value corresponding to the number of pixels in the image:

print(alpha) // [0, 0, 0, 0, 0 ... 409600]

A histogram won't give you the dominant color from a visual perspective: an image which is half yellow {1,1,0} and half black {0,0,0} will give the same results as an image which is half red {1,0,0} and held green {0,1,0} .

Hope this helps,

Simon

Answer 3

One problem with the histogram approach is that you lose correlation between the color channels. That is, half your image could be magenta and half yellow. You will find a red histogram that is all in the 1.0 bin, but the blue and green bins would be evenly split between 0.0 and 1.0 with nothing in between. Even though you can be quite sure that red is bright, you won't be able to say much about what the blue and green component should be for the "predominant color"

You could use a 3D histogram with 2**(8+8+8) bins, but this is quite large and you will find the signal is quite sparse. By happenstance three pixels might land in one bin and have no two the same elsewhere, even though many users could tell you that there is a predominant color and it has nothing to do with that pixel.

You could make the 3D histogram a lot lower resolution and have (for example) just 16 bins per color channel. It is much more likely that bins will have a statistically meaningful population count this way. This should give you a starting point to find a mean for a local population of pixels in that bin. If each bin had a count and a {R,G,B} sum, then you could quickly find the mean color for pixels in that bin once you had identified the most popular bins. This method is still subject to some influence from the histogram grid. You will be more likely to identify colors in the middle of a grid cell than at the edges. Populations may span multiple grid cells. Something like kmeans might be another method.

If you just want predominant hue, then conversion to a color space like HSV followed by a histogram of hue would work.

I'm not aware of any filters in vImage, CI or MetalPerformanceShaders to do these things for you. You can certainly write code in either the CPU or Metal to do it without a lot of trouble.