OpenCV dll calls from Unity3D lead to FPS drop

Question

I want to recognize ArUco Marker in Unity3D and attach a GameObject to their position. I know there are packages in the Asset Store, but as other people got it working I was looking for an free existing solution or try it by myself.

ArucoUnity The detection worked like a charm, but Unity crashes when accessing the data of the rvecs , tvecs . The error somewhere occur in the Get(int i) in the Vec3d class, when the c++ method au_cv_Vec3d_get(CppPtr, i, CppPtr); is called

OpenCv plus Unity This implementation seems not to be complete as there exist no function to EstimatePoseSingleMarker or similar to get the rvecs and tvecs . Also the last updated was Jan 2019.

As Unity (C#) provides the possibility to access unmanaged code, I followed this great tutorial and for the begging I was able to forward the cv::VideoCaputre stream to Unity. The only problem that occured was a FPS drop to around 35-40 whereas normally I get around 90-100.

The C# code:

void Update()
{
    MatToTexture2D();
    image.texture = tex;
}

void MatToTexture2D()
{
    OpenCVInterop.GetRawImageBytes(pixelPtr);
    //Update the Texture2D with array updated in C++
    tex.SetPixels32(pixel32);
    tex.Apply();
}

The C++ code:

extern "C" void __declspec(dllexport) __stdcall  GetRawImageBytes(unsigned char* data)
{
    _capture >> _currentFrame;

    cv::Mat resizedMat(camHeight, camWidth, _currentFrame.type());
    cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);

    //Convert from RGB to ARGB 
    cv::Mat argb_img;
    cv::cvtColor(resizedMat, argb_img, cv::COLOR_BGR2BGRA);
    std::vector<cv::Mat> bgra;
    cv::split(argb_img, bgra);
    std::swap(bgra[0], bgra[3]);
    std::swap(bgra[1], bgra[2]);
    std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}

The cause seems to be the first line _capture >> _currentFrame; , but as the others projects have to do the same (at least I guess so), I wonder if there is another reason. If I don't manage to fix this issue I have to look for alternative approaches.

Answer 1

Just adding to / building on Mars' answer :

For the threading problem I would actually use a thread-save ConcurrentStack<Color32[]> . A stack is "last-in | first-out" so that the first item returned is always the last image data added by the thread.

• The thread uses Push(pixel32) to add a new entry with image data
• in Update you only use the latest entry ( TryPop ) for updating the texture.
• The rest you ignore ( Clear ).

So something like

// the OpenCV thread will add(push) entries
// the Unity main thread will work on the entries
private ConcurrentStack<Color32[]> stack = new ConcurrentStack<Color32[]>();

public RawImage image;
public Texture2D tex;

private Thread thread;

void Start()
{
    // Wherever you get your tex from
    tex = new Texture2D(...);

    // it should be enough to do this only once
    // the texture stays the same, you only update its content
    image.texture = tex;
}

// do things in OnEnable so everytime the object gets enabled start the thread
void OnEnable()
{
    stack.Clear();

    if(thread != null)
    {
        thread.Abort();
    }

    thread = new Thread(MatToTexture2D);
    thread.Start();
}

void Update()
{
    // here in the main thread work the stack
    if (stack.TryPop(out var pixels32))
    {
        // Only use SetPixels and Apply when really needed
        tex.SetPixels32(pixels32);
        tex.Apply();
    }

    // Erase older data
    stack.Clear();
}

// Make sure to terminate the thread everytime this object gets disabled
private void OnDisable()
{
    if(thread == null) return;

    thread.Abort();
    thread = null;
}

// Runs in a thread!
void MatToTexture2D()
{
    while(true)
    {
        try
        {
            // Do what you already have
            OpenCVInterop.GetRawImageBytes(pixelPtr);

            // However you convert the pixelPtr into Color32
            Color32[] pixel32 = GetColorArrayFromPtr(pixelPtr);

            // Now add this data to the stack
            stack.Push(pixel32);
        }
        catch (ThreadAbortException ex) 
        { 
            // This exception is thrown when calling Abort on the thread
            // -> ignore the exception since it is produced on purpose
        } 
    }
}

Answer 2

If I recall correctly, the C++ call to get an image ( _capture >> _currentFrame; ) is blocking/synchronous, meaning your code won't continue until it actually retrieves the image. You probably want to run your MatToTexture2D code asynchronously. ※This will mean that your frame rate will be higher than your image retrieval rate.

Have your MatToTexture2D function run continuously as needed, updating tex . Then just continue to set your texture to the latest tex , which may be the same value 2-3 frames in a row.

---

Edit: @derHugo's answer is much more solid for the programming side, so I'll hide that part. The basic issue is explained above, and derHugo's work-around is much better than my pseudo-code:)