Hololens - Unity c# : projecting a point set from camera's plan to unity coordinate system?

Question

I am currently working on an app for Hololens that allows the user to take multiple frames of a scene from different points of view (let's say a box for example) and convert these to a point set representing the object. Getting the depth bitmap for each frame is working well but my problem comes when I want to transform the point set to the Unity coordinate System...

Each time the user takes a capture, I get the depth bitmap, the frame's coordinate system, the projection transform matrix, the camera view transform matrix and the transform matrix from camera's coordinate system to unity's coordinate system.

// Get the spatial coordinates system from the mediaFrameReferecence
var coordinateSystem = mediaFrameReference.CoordinateSystem;

// Get the projection Transform matrix
object n;
mediaFrameReference.Properties.TryGetValue(projectionTransformGuid, out n);

// ByteArrayMatrix(byte[]) is a method a developped which works fine
projectionTransformMatrix = ByteArrayToMatrix(n as byte[]);

/// HERE n is a byte[48] but I'm expecting byte[64] like m below

// Get the view transform then invert it
byte[] m = mediaFrameReference.Properties[viewTransformGuid] as byte[];
cameraViewTransformMatrix = ConvertByteArrayToMatrix4x4(m);

// Get the camera to world transfrom
cameraToWorldTransformMatrix = (System.Numerics.Matrix4x4)coordinateSystem.TryGetTransformTo(rootSpatialCoordinateSystem);

// ... Doing Some stuff ...

After having those, for each pixel I save my point to a .obj file with the format : vxyzrgb in SavePoint method below :

private static Vector2 PixelToWorldCoordonate(int u, int v)
{
    float x = a * (u - width / 2) + b * (v - height / 2);
    float y = c * (u - width / 2) + d * (v - height / 2);
    return new Vector2(x, y);
}

/// This method is to project a pixel to unity coordinate system
private String SavePoint((int x, int y, byte* inputRowBytes,
                          float depthScale, float minReliableDepth, float maxReliableDepth,
                          System.Numerics.Matrix4x4 cameraViewTransformMatrix, 
                          System.Numerics.Matrix4x4 cameraToWorldTransformMatrix, 
                          double r, double g, double b) 
{
    if (depth < 2)
    {
        string mes = "";
        Vector2 realPoint = PixelToWorldCoordonate(x, y);
        System.Numerics.Vector3 point3d = new System.Numerics.Vector3(realPoint.x, realPoint.y, 1.0f);
        point3d *= depth;
        System.Numerics.Vector3 position = System.Numerics.Vector3.Transform(point3d, cameraViewTransformMatrix);

        // Saving the point's position and color
        return "v " + position.X.ToString() + " " + position.Y.ToString() + " " + position.Z.ToString()
            + " " + r.ToString() + " " + g.ToString() + " " + b.ToString() + Environment.NewLine;
    }
}

With my actual code, I get coherent points sets for each capture apart of others, but here is the probleme : When taking multiple captures, the points sets seem to be in different plans (see screenshot below) :

In this image the red part is the point set for the first capture and the green one for a second capture after moving 50 cm on right.. the circled areas are the object I'm looking at when taking the picture. I was hoping that the object would stay in same spot as I move around but I may be wrong after all...

If anyone has even a tiny idea of what I could be missing

Answer 1

尝试在统一坐标系的原点上添加一个 World 锚点，并映射与框架参考相关的位置。

Answer 2

Aside from what Herdnando already said, the problem might be that the cameraViewTransformMatrix only transforms between camera and point. So when the camera moves, the reference coordinate system moves with it and the point clouds do not align. You also have to apply your cameraToWorldTransformMatrix to get everything into your root coordinate system.