[英]Convert OpenCV Mat to Texture2D?
Meta Context: 元上下文:
I'm currently working on a game that utilizes opencv as a substitute for ordinary inputs (keyboard, mouse, etc...). 我目前正在开发一款使用opencv替代普通输入(键盘,鼠标等)的游戏。 I'm using Unity3D's C# scripts and opencv in C++ via DllImports. 我正在通过DllImports在C ++中使用Unity3D的C#脚本和opencv。 My goal is to create an image inside my game coming from opencv. 我的目标是在游戏中创建来自opencv的图像。
Code Context: 代码上下文:
As done usually in OpenCV, I'm using Mat to represent my image. 像在OpenCV中通常所做的那样,我使用Mat来代表我的图像。 This is the way that I'm exporting the image bytes: 这是我导出图像字节的方式:
cv::Mat _currentFrame;
...
extern "C" byte * EXPORT GetRawImage()
{
return _currentFrame.data;
}
And this is how i'm importing from C#: 这就是我从C#导入的方式:
[DllImport ("ImageInputInterface")]
private static extern IntPtr GetRawImage ();
...
public static void GetRawImageBytes (ref byte[] result, int arrayLength) {
IntPtr a = GetRawImage ();
Marshal.Copy(a, result, 0, arrayLength);
FreeBuffer(a);
}
Judging by the way I understand OpenCV, I expect the byte array to be structured in this way when serialized in a uchar pointer: 从我对OpenCV的了解来看,我期望在uchar指针中进行序列化时,字节数组将以这种方式构造:
b1, g1, r1, b2, g2, r2, ...
I'm converting this BGR array to a RGB array using: 我正在使用以下方法将此BGR数组转换为RGB数组:
public static void BGR2RGB(ref byte[] buffer) {
byte swap;
for (int i = 0; i < buffer.Length; i = i + 3) {
swap = buffer[i];
buffer[i] = buffer[i + 2];
buffer[i + 2] = swap;
}
}
Finally, I'm using Unity's LoadRawTextureData
to load the bytes to a texture: 最后,我使用Unity的LoadRawTextureData
将字节加载到纹理:
this.tex = new Texture2D(
ImageInputInterface.GetImageWidth(),
ImageInputInterface.GetImageHeight(),
TextureFormat.RGB24,
false
);
...
ImageInputInterface.GetRawImageBytes(ref ret, ret.Length);
ImageInputInterface.BGR2RGB(ref ret);
tex.LoadRawTextureData(ret);
tex.Apply();
Results: 结果:
The final image seems to be scattered in someway, it resembles some shapes, but it seems to triple the shapes as well. 最终的图像似乎以某种方式散落,类似于某些形状,但看起来也使形状增加了三倍。 This is me holding my hand in front of the camera: 这是我在镜头前握着的手:
[Me, my hand and the camera] [我,我的手和相机]
Doing some tests, I concluded that I decoded the channels correctly, since, using my phone to emit RGB light, I can reproduce the colors from the real world: 做一些测试,我得出的结论是我正确解码了通道,因为使用手机发出RGB光,我可以从真实世界中再现颜色:
[Red Test] [红色测试]
[Blue Test] [蓝色测试]
[Green Test] [绿色测试]
There are also some strange lines in the image: 图像中还有一些奇怪的行:
[Spooky Lines] [怪异的线条]
There is also my face to compare these images to: 我还有一张脸可将这些图像与以下图像进行比较:
[My face in front of the camera] [我在镜头前的脸]
Questions: 问题:
Since I'm able to correctly decode the color channels, what have I assumed wrong in decoding the OpenCV array? 由于我能够正确解码颜色通道,因此在解码OpenCV阵列时我认为有什么错误? It's that I don't know how the Unity's LoadRawTextureData
works, or have I decoded something in the wrong way? 是因为我不知道Unity的LoadRawTextureData
是如何工作的,或者我是否以错误的方式解码了某些内容?
How is the OpenCV Mat.data
array structured? OpenCV Mat.data
数组的结构如何?
UPDATE 更新
Thanks to @Programmer, his solution worked like magic. 感谢@Programmer,他的解决方案像魔术一样工作。
[Me Happy] [我开心]
I changed his script a little, there was no need to do some stuff. 我稍微修改了他的脚本,不需要做任何事情。 And in my case i needed to use BGR2RGBA, not RGB2RGBA: 在我的情况下,我需要使用BGR2RGBA,而不是RGB2RGBA:
extern "C" void EXPORT GetRawImage( byte *data, int width, int height )
{
cv::Mat resizedMat( height, width, _currentFrame.type() );
cv::resize( _currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC );
cv::Mat argbImg;
cv::cvtColor( resizedMat, argbImg, CV_BGR2RGBA );
std::memcpy( data, argbImg.data, argbImg.total() * argbImg.elemSize() );
}
Use SetPixels32
instead of LoadRawTextureData
. 使用SetPixels32
代替LoadRawTextureData
。 Instead of returning the array data from C++, do that from C#. 与其从C ++返回数组数据,不如从C#中返回数组数据。 Create Color32
array and pin it in c# with GCHandle.Alloc
, send the address of the pinned Color32
array to C++, use cv::resize
to resize the cv::Mat
to match the size of pixels sent from C#. 创建Color32
数组,然后使用GCHandle.Alloc
将其固定在c#中,将固定的Color32
数组的地址发送到C ++,使用cv::resize
调整cv::Mat
的大小以匹配从C#发送的像素大小。 You must do this step or expect some error or issues. 您必须执行此步骤,否则可能会出现一些错误或问题。
Finally, convert cv::Mat
from RGB to ARGB then use std::memcpy
to update the array from C++. 最后,将cv::Mat
从RGB转换为ARGB,然后使用std::memcpy
从C ++更新数组。 The SetPixels32
function can then be used to load that updated Color32
array into Texture2D
. 然后可以使用SetPixels32
函数将更新后的Color32
数组加载到Texture2D
。 This is how I do it and it has been working for me without any issues. 这就是我的操作方式,它一直为我工作,没有任何问题。 There might be other better ways to do it but I have never found one. 可能还有其他更好的方法可以执行此操作,但我从未找到一种方法。
C++: C ++:
cv::Mat _currentFrame;
void GetRawImageBytes(unsigned char* data, int width, int height)
{
//Resize Mat to match the array passed to it from C#
cv::Mat resizedMat(height, width, _currentFrame.type());
cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);
//You may not need this line. Depends on what you are doing
cv::imshow("Nicolas", resizedMat);
//Convert from RGB to ARGB
cv::Mat argb_img;
cv::cvtColor(resizedMat, argb_img, CV_RGB2BGRA);
std::vector<cv::Mat> bgra;
cv::split(argb_img, bgra);
std::swap(bgra[0], bgra[3]);
std::swap(bgra[1], bgra[2]);
std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}
C#: C#:
Attach to any GameObject with a Renderer
and you should see the cv::Mat
displayed and updated on that Object every frame. 使用Renderer
附加到任何GameObject上,您应该看到cv::Mat
每帧在该对象上显示和更新。 Code is commented if confused: 如果混淆,则对代码进行注释:
using System;
using System.Runtime.InteropServices;
using UnityEngine;
public class Test : MonoBehaviour
{
[DllImport("ImageInputInterface")]
private static extern void GetRawImageBytes(IntPtr data, int width, int height);
private Texture2D tex;
private Color32[] pixel32;
private GCHandle pixelHandle;
private IntPtr pixelPtr;
void Start()
{
InitTexture();
gameObject.GetComponent<Renderer>().material.mainTexture = tex;
}
void Update()
{
MatToTexture2D();
}
void InitTexture()
{
tex = new Texture2D(512, 512, TextureFormat.ARGB32, false);
pixel32 = tex.GetPixels32();
//Pin pixel32 array
pixelHandle = GCHandle.Alloc(pixel32, GCHandleType.Pinned);
//Get the pinned address
pixelPtr = pixelHandle.AddrOfPinnedObject();
}
void MatToTexture2D()
{
//Convert Mat to Texture2D
GetRawImageBytes(pixelPtr, tex.width, tex.height);
//Update the Texture2D with array updated in C++
tex.SetPixels32(pixel32);
tex.Apply();
}
void OnApplicationQuit()
{
//Free handle
pixelHandle.Free();
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.