RGB to HIS giving NaN for certain RGB values

Question

I have a function that calculates hue for a given rgb value. For some values, the acos returns NaN because the argument is slightly larger than 1. This code follows most of the formulas I found online but I can't figure out why NaN is popping up.

Examples are

40 28 28
40 28 28
40 28 28
49 25 25
46 34 34
40 28 28
42 24 24
42 24 24
40 22 22
40 22 22

#include <math.h>

double hue(unsigned char r, unsigned char g, unsigned char b) {
  double rn = (double) r / (r + g + b);
  double gn = (double) g / (r + g + b);
  double bn = (double) b / (r + g + b);

  if (rn == gn && gn == bn) {
    return 0;
  }

  double h = acos((rn - gn + rn - bn) / (2.0 * sqrt((rn - gn) * (rn - gn) + (rn - bn) * (gn - bn))));

  // issue with the argument of acos being just slightly bigger than 1. Approximate to 0
  if (isnan(h)) {
    h = 0;
  }

  if (b > g) {
    return (2 * M_PI - h) * (180.0 / M_PI);
  } else {
    return h * (180.0 / M_PI);
  }
}

Answer 1

I can't figure out why NaN is popping up

And the answer is there in your question:

because the argument is slightly larger than 1

---

The correct question would be in this case:

How should I make sure that the argument to acos is not bigger than 1?

For that, you need to analyze your formula and see where the problem comes from. It is possible that the sqrt and the division play a role.

The easiest hack (might be bad for your application) is to verify first if the argument is bigger than 1, and if yes, truncate it to 1.

---

Please notice that you loose extra precision calculating rn , gn , bn . The argument to sqrt might better be calculated as:

a = (r-g)*(r-g)+(r-b)*(g-b);
b = (r+g+b)*(r+g+b);
argument = a/b;

Play with it and see what happens.

---

Also, the sequence

// issue with the argument of acos being just slightly bigger than 1. Approximate to 0
if (isnan(h)) {
    h = 0;
}

should be used before calling acos , and using the (precalculaed) argument of acos instead of the calculated h .