Floating point error in HSV conversion

Question

I have a function for converting vectors in RGB colorspace to HSV - the routine passes all tests and works the vast majority of the time. However, crash reporting indicates that it occasionally fails (0.0001% of calls or less.) But, since this function is called so much this accounts for a substantial portion of the total crashes in the software.

The relevant portion of the code is as follows:

glm::vec4 RGBtoHSV(glm::vec4 rgb)
{
    vec4 hsv = vec4(0,0,0,rgb.w);

    float r = rgb.x;
    float g = rgb.y;
    float b = rgb.z;
    float M = std::max(std::max(r,g),b);
    float m = std::min(std::min(r,g),b);
    float c = M - m;

    //calc hue
    float hp = 0;
    if (c == 0)
        hp = 0;
    else if (M == r)
        hp = glm::mod(((g-b)/c), 6.0f);
    else if (M == g)
        hp = ((b-r)/c) + 2;
    else if (M == b)
        hp = ((r-g)/c) + 4;
    else
        EXPECT(false);
    float h = hp * 60;

    ...

glm::vec4 is a reference to the GLM library, but the type is basically just this:

struct vec4 {
    union {
        struct { float x, y, z, w; }
        struct { float r, g, b, a; }
    }
}

The failing line is EXPECT(false); -- which is just asserting that we always reach one of the upper cases, and the question is how can control ever reach there?

Answer 1

Assuming that these are IEEE-754 floating point numbers then you might have a problem comparing with NaN . If you are using a different floating point representation then the rest of my answer might not be relevant.

It might sound a bit strange but any comparison with NaN returns False. So even if you are attempting to do NaN == Nan you get false. MSDN link .

If you test for NaN in the part of your code where the assertion is raised then this is likely the problem. You can do this by using the isnan() found in <cmath> see http://en.cppreference.com/w/cpp/numeric/math/isnan .

I quickly put together some code to demonstrate, please let me know if the same happens on your system:

#include <iostream>
#include <cmath>
int main(){
    float r = 0.0/0;
    std::cout << "r = " << r <<  std::endl;
    float g = 7.0;
    float b = 0.0;
    float M = std::max(std::max(r,g),b);
    float m = std::min(std::min(r,g),b);
    float c = M - m;

    //calc hue
    float hp = 0;
    if (c == 0)
        std::cout << "c == 0" << std::endl;
    else if (M == r)
        std::cout << "M == r" << std::endl;
    else if (M == g)
        std::cout << "M == g" << std::endl;
    else if (M == b)
        std::cout << "M == b" << std::endl;
    else
        std::cout << "oops" << std::endl;
        if(std::isnan(M))
            std::cout << "NaN encountered" << std::endl;

}

On my machine compiling with gcc this gives the following output:

r = -nan
oops
NaN encountered

So it seems somewhat likely that there is a 0.0/0 going on somewhere else in the code.