Kinect v2 - 同步深度和颜色帧

Question

I am currently looking for a stereoscopic camera for a project and the Kinect v2 seems to be a good option. However, since it's quite an investment to me, I need to be sure it meets my requirements, the main one being a good synchronization of the different sensors.

Apparently there is no hardware synchronization of the sensors, and I get many versions about the software part:

1. Some posts where people complain about lag between the 2 sensors, and many others asking for a way to synchronize the sensors. Both seem to have strange workarounds and no "official", common solution emerges from the answers.

2. Some posts about a MultiSourceFrame class, which is part of Kinect SDK 2.0. From what I understand, this class enables you to retrieve the frame of all the sensors (or less, you can choose which sensors you want to get the data from) at a given time. Thus, you should be able, for a given instant t, to get the output of the different sensors and make sure these outputs are synchronized.

So my question is, is this MultiSourceFrame class doing exactly what I mean it does? And if yes, why is it never proposed as a solution? It seems the posts of the 1st category are from 2013, so before the release of the SDK 2.0. However, MultiSourceFrame class is supposed to replace the AllFramesReady event of the previous versions of the SDK, and AllFramesReady wasn't suggested as a solution either.

Unfortunately the documentation doesn't provide much information about how it works, so I'm asking here in case someone would have already used it. I'm sorry if my question seems stupid, but I would like to be sure before purchasing such a camera.

Thank you for your answers! And feel free to ask for more details if needed :)

Answer 1

There was a discussion about that in an libfreenect2 issue , where someone specifically mentioned a 6.25 millisecond lag between RGB and depth frame when using the MultiSourceFrameReader :

The RelativeTime of the ColorFrame seems to always lags 6.25 or 6.375 ms behind the RelativeTime of the DepthFrame, InfraredFrame, BodyFrame, BodyIndexFrame. Meanwhile, the RelativeTime always matches among DepthFrame, InfraredFrame, BodyFrame, and BodyIndexFrame.

In my own experiments, I got the same results. But that's only based on the frame's timestamps. These timestamps come from the Kinect v2 device directly, so it's unlikely, but still be possible that they are not 100% correct.

So, while there is a lag between depth and RGB frames, even when using MultiSourceFrameReader , it's most likely small enough so you can ignore it.

As for the usage of MultiSourceFrame / MultiSourceFrameReader , it's pretty simple once you got used to the Kinect v2 SDK:

m_pKinectSensor->OpenMultiSourceFrameReader(
    FrameSourceTypes::FrameSourceTypes_Depth | FrameSourceTypes::FrameSourceTypes_Color,
    &m_pMultiSourceFrameReader);

// get "synced" frame
IMultiSourceFrame* pMultiSourceFrame = NULL;
m_pMultiSourceFrameReader->AcquireLatestFrame(&pMultiSourceFrame);

// get depth frame
IDepthFrameReference* pDepthFrameReference = NULL;
pMultiSourceFrame->get_DepthFrameReference(&pDepthFrameReference);
IDepthFrame* pDepthFrame = NULL;
pDepthFrameReference->AcquireFrame(&pDepthFrame);

// get RGB frame
IColorFrameReference* pColorFrameReference = NULL;
pMultiSourceFrame->get_ColorFrameReference(&pColorFrameReference);
IColorFrame* pColorFrame = NULL;
pColorFrameReference->AcquireFrame(&pColorFrame);

// ... now use both frames

You can find more details in the CoordinateMapping Basic sample, once you installed the Kinect v2 SDK.

Answer 2

I've only used the MS SDK but I figure the rules apply. The reason why Relative time is the same for all the above streams is because all of the above are created out of the IR frame, thus they are all dependent on it. The color frame is not as it comes from a different camera. As for RelativeTime, it's basically a TimeSpan(in C# terms) which describes something akin a delta time between frames in Kinect's own runtime clock. It's probably created by the Kinect Service which grabs the raw input from the sensor, sends the IR to GPU for expansion into Depth(which is actually an averaging of several frames), Body and BodyFrame(and LongExposureIR) and then gets them back and gets the data back in the CPU to distributed to all the registered listeners(aka different Kinect v2 apps/instances). Also read in an MSDN forum a reply by an MVP who said MS cautioned them from using RelativeTime for anything other than delta time usage. So I don't know if you can actually use it for manual synchro between separate streams(ie without the use of MultiSourceFrameReader) with confidence.