Get RGB values of each 3D point in Kinect v2

Question

I am getting confused with the Kinect v2 CoordinateMapper API. I want to get RGB values of each 3D point aka CameraSpacePoint in Kinect v2.

Please see below the code snippet:

var depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
var colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();

var depthWidth = depthFrame.FrameDescription.Width;
var depthHeight = depthFrame.FrameDescription.Height;

ushort[] depthData = new ushort[depthWidth * depthHeight];
CameraSpacePoint[] camerapoints = new CameraSpacePoint[depthData.Length];
ColorSpacePoint[] colorpoints = new ColorSpacePoint[depthData.Length];

depthFrame.CopyFrameDataToArray(depthData);
this.coordinateMapper.MapDepthFrameToCameraSpace(depthData, camerapoints);
this.coordinateMapper.MapDepthFrameToColorSpace(depthData, colorpoints);

The 3D points are stored in camerapoints variable. I want to the RGB values of each camerapoints . In other words, please see below pesudo code:

RGBPoint[] rgbpoints = new RGBPoint[depthData.Length];
RGBPoint rgbpoint = rgbpoints[0];
int red   = rgbpoint.r
int green = rgbpoint.g
int blue  = rgbpoint.b

As always, thank you very much. I really appreciate your kind response.

Answer 1

The colorpoints array contains the (x, y) index that corresponds to each pixels in the depthData array. From here, we can just get the color frame data (from colorFrame ) at each specific (x, y) index indicated by colorpoints[index] .

var colorWidth = colorFrame.FrameDescription.Width;
var colorHeight = colorFrame.FrameDescription.Height;

// Assuming BGRA format here
byte[] pixels = new byte[colorWidth * colorHeight * 4];
colorFrame.CopyFrameDataToArray(pixels);

byte[] bgraPoints = new byte[depthWidth * depthHeight * 4];

for (var index = 0; index < depthData.Length; index++)
{
    var u = colorpoints[index].X;
    var v = colorpoints[index].Y;
    if (u < 0 || u >= colorWidth || v < 0 || v >= colorHeight) continue;
    var pixelsBaseIndex = v * colorWidth + u;
    Array.Copy(pixels, 4 * pixelsBaseIndex, bgraPoints, 4 * index, 4);
}

var index = 0;
var red = bgraPoints[4 * index + 2];
var green = bgraPoints[4 * index + 1];
var blue = bgraPoints[4 * index];