Hololens video stream with spatial data

Question

I'm using the hololens and I'm trying to save a video stream with the world/projection matrices avaiable. I've been trying to just take a sequence of pictures and save the data, but I can't find a way to save the image and the matrices.

When saving to disk, there is no option to get the photocaptureframe (which contains the matrix data), when saving to memory, I seem to not be able to save the image to disk. I tried using the following methode, but this seemed to crash my unity program:

List<byte> imageBufferList = new List<byte>();
photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);
byte[] myArrayImage = imageBufferList.ToArray();

And then use this to convert the byte array:

using (MemoryStream mStream = new MemoryStream(byteArrayIn))
return Image.FromStream(mStream);

After which I save the result.

When I remove the memorystream thing, the program doesn't crash (but it doesn't save my image either).

I've been looking all over the internet but there are a lot of vague statements about it a) not beeing possible b) using the memorystream (but that crashes)

Any suggestions? If anyone knows a way to save all the the matrix (projection and world) data per frame and the corresponding frame for a video stream, it would be a great help.

Edit: I also tried to look into https://github.com/VulcanTechnologies/HoloLensCameraStream but this seems to give problems with newer Unity versions. Any remarks about this?

To clarify my end goal: When filming, the program should save all frames and the corresponding matrices, for example:

Frame_01, Frame_02, Frame_03, ... (.jpg/png/...)

World_matrix_01, World_matrix_02, ... (.txt)

Projection_matrix_01, Projection_matrix_02,... (.txt)

Answer 1

Edit: I also tried to look into https://github.com/VulcanTechnologies/HoloLensCameraStream but this seems to give problems with newer Unity versions. Any remarks about this?

I used it and it worked very well in Unity 2018.3.13f1. You can only test in Hololens out of debug mode.