Kinect v2 红外传感器和 RGB 图像的对齐总是略微关闭

Question

I'm using the official Kinect SDK 2.0 and Emgu CV in order to recognize the colors of a Rubik's Cube.

At first I use Canny Edge Extraction on the Infrared Camera since it handles different lightning conditions better than the RGB Camera and is much better to detect contours.

Then I use this code to convert the coordinates of the infrared sensor to the ones of the RGB camera. As you can see the in the picture they are still off from what I am looking for. Since I already use the official KinectSensor.CoordinateMapper.MapDepthFrameToColorSpace I don't know how else I can improve the situation.

using (var colorFrame = reference.ColorFrameReference.AcquireFrame())
using (var irFrame = reference.InfraredFrameReference.AcquireFrame())
{
    if (colorFrame == null || irFrame == null)
        return;

    // initialize depth frame data 
    FrameDescription depthDesc = irFrame.FrameDescription;

    if (_depthData == null)
    {
        uint depthSize = depthDesc.LengthInPixels;
        _depthData = new ushort[depthSize];
        _colorSpacePoints = new ColorSpacePoint[depthSize];

         // fill Array with max value so all pixels can be mapped
         for (int i = 0; i < _depthData.Length; i++)
         {
             _depthData[i] = UInt16.MaxValue;
         }
         // didn't work so well with the actual depth-data
         //depthFrame.CopyFrameDataToArray(_depthData);

        _sensor.CoordinateMapper.MapDepthFrameToColorSpace(_depthData, _colorSpacePoints);
    }
}

This is a helper-function I created in order to convert Point-Arrays in Infrared-Space to Color-Space

public static System.Drawing.Point[] DepthPointsToColorSpace(System.Drawing.Point[] depthPoints, ColorSpacePoint[] colorSpace){
        for (int i = 0; i < depthPoints.Length; i++)
        {
            // 512 is the width of the depth/infrared image
            int index = 512 * depthPoints[i].Y + depthPoints[i].X;

            depthPoints[i].X = (int)Math.Floor(colorSpace[index].X + 0.5);
            depthPoints[i].Y = (int)Math.Floor(colorSpace[index].Y + 0.5);
        }
        return depthPoints;
    }

Answer 1

We can solve this problem by transforming infrared image coordinates to color image coordinates with 2 quadrilateral mapping.

A quadrilateral Q(x1,y1,x2,y2,x3,y3,x4,y4) in an infrared image, similarly, it's mapping quadrilateral Q'(x1',y1',x2',y2',x3',y3',x4',y4') in the corresponding color image.

We can write the above mapping in form of equation as follows:

Q'= Q*A

where, A is a 3 X 3 matrix with coefficients a11, a12, a13, a21,.., a33;

The formula to obtain the coefficients are listed as follows:

x1=173; y1=98; x2=387; y2=93; x3=395; y3=262; x4=172; y4=264;  
x1p=787; y1p=235;  x2p=1407; y2p=215; x3p=1435; y3p=705;  x4p=795; y4p=715; 
tx=(x1p-x2p+x3p-x4p)*(y4p-y3p)-(y1p-y2p+y3p-y4p)*(x4p-x3p);
ty=(x2p-x3p)*(y4p-y3p)-(x4p-x3p)*(y2p-y3p);
a31=tx/ty;
tx=(y1p-y2p+y3p-y4p)*(x2p-x3p)-(x1p-x2p+x3p-x4p)*(y2p-y3p);
ty=(x2p-x3p)*(y4p-y3p)-(x4p-x3p)*(y2p-y3p);
a32=tx/ty;
a11=x2p-x1p+a31*x2p;
a12=x4p-x1p+a32*x4p;
a13=x1p;
a21=y2p-y1p+a31*y2p;
a22=y4p-y1p+a32*y4p;
a23=y1p;
a33=1.0;

Answer 2

Its because its not the same camera the camera that retrieves the depth data and the one that retrieves color data. So you should apply a correction factor to displace the depth data. Its a factor that is almost constant but its related to the distance. I've got no code for you, but its something you can calculate yourself.