IndexOutOfRange Exception while trying to get color values from Kinect v2 in C#

Question

I am trying to get the RGB values of each CameraSpacePoint in Kinect v2. While trying to access converted frame data, the code is throwing IndexOutOfRangeException . I found out that following lines are throwing this error:

byte red   = pixels[4 * pixelsBaseIndex + 0];
byte green = pixels[4 * pixelsBaseIndex + 1];
byte blue  = pixels[4 * pixelsBaseIndex + 2];
byte alpha = pixels[4 * pixelsBaseIndex + 3];

Please see the complete code below:

int depthWidth = kinectSensor.DepthFrameSource.FrameDescription.Width;
int depthHeight = kinectSensor.DepthFrameSource.FrameDescription.Height;
int colorWidth = kinectSensor.ColorFrameSource.FrameDescription.Width;
int colorHeight = kinectSensor.ColorFrameSource.FrameDescription.Height;

ushort[] depthData = new ushort[depthWidth * depthHeight];
CameraSpacePoint[] cameraPoints = new CameraSpacePoint[depthData.Length];
ColorSpacePoint[] colorPoints = new ColorSpacePoint[depthData.Length];

// Assuming RGBA format here
byte[] pixels = new byte[colorWidth * colorHeight * 4];

depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();

if ((depthFrame == null) || (colorFrame == null))
{
    return;
}

depthFrame.CopyFrameDataToArray(depthData);
coordinateMapper.MapDepthFrameToCameraSpace(depthData, cameraPoints);
coordinateMapper.MapDepthFrameToColorSpace(depthData, colorPoints);

// We are using RGBA format
colorFrame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Rgba);

for (var index = 0; index < depthData.Length; index++)
{
    var u = colorPoints[index].X;
    var v = colorPoints[index].Y;

    if (u < 0 || u >= colorWidth || v < 0 || v >= colorHeight) continue;

    int pixelsBaseIndex = (int)(v * colorWidth + u);
    try
    {
        byte red = pixels[4 * pixelsBaseIndex + 0];
        byte green = pixels[4 * pixelsBaseIndex + 1];
        byte blue = pixels[4 * pixelsBaseIndex + 2];
        byte alpha = pixels[4 * pixelsBaseIndex + 3];
    }
    catch (IndexOutOfRangeException ex)
    {
        int minValue = 4 * pixelsBaseIndex;
        int maxValue = 4 * pixelsBaseIndex + 3;
        Console.WriteLine((minValue > 0) + ", " + (maxValue < pixels.Length));
    }
}

The code looks fine however I am not sure what am I missing here! How to avoid IndexOutOfRangeException exception? Any suggestions, please?

Answer 1

First thing pixelsBaseIndex =< 2070601 (2070601 = 1919 * 1079)

2074817 is an invalid value for the pixelsBaseIndex .

This error could be somewhere here.

if (u < 0 || u >= colorWidth || v < 0 || v >= colorHeight) continue;

int pixelsBaseIndex = (int)(v * colorWidth + u);

Best thing is to console log u, v, colorWidth, colorHeight and validate them.

EDIT

As i suspected, it is caused by the Floating-Point Arithmetic. Mapping one frame to another frame needs complex matrix calculations, thus u,v get floating point values. Keep in mind that u,v are the pixel coordinates of the color space. They must be non-negative integers.

My recommended solution is

...
    int u = (int) Math.Floor(colorPoints[index].X);
    int v = (int) Math.Floor(colorPoints[index].Y);

    if (u < 0 || u >= colorWidth || v < 0 || v >= colorHeight) continue;
...

FYI

Check out following lines of code to understand why u, v pixel coordinates getting floating point values. The decision, taking the floor or discarding the invalid values, is depending on the developer since in either cases, you are loosing some sort of information. For example, taking the floor means you are taking the color of the nearest pixel.

In my personal option, out putting more precise value (thus floating point value) is important for developers who do more calculations on the outputted values.For example, point cloud registration.

depth_to_color

calculating x offset for rgb image based on depth value