Copying a small image into the camera frame with OpenCV for Android

Question

I'm currently working on a program that should draw a small image onto the camera frame. With Android OpenCV, you do have the following function to process a frame:

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

      Mat rgba = inputFrame.rgba();

      mDetector.setFrame(rgba);
      mDetector.processFrame();

      return rgba;
}

Where the Mat rgba then gets displayed on the screen. My Detector should now process the frame rgba (change it). Here is the relevant code:

public void processFrame() {

    // (1) Doesn't work
    Rect roi = new Rect(0, 0, 100, 100);
    Mat submat = mOutputFrame.submat(roi);
    Mat image =  new Mat(100, 100, CvType.CV_8UC3, new Scalar(0,0,0));
    image.copyTo(submat);

    // (2) Does work 
    // --- mComparatorImage is the same size as mOutputFrame.
    // --- mComparatorImage is 8bit greyscale, mOutputFrame is the rgba CameraFrame
    mComparatorImage = mComparatorHolder.getCurrentImage();
    mComparatorImage.copyTo(mOutputFrame);

    // (3) Should work (but doesn't)
    Imgproc.resize(mComparatorImage, mResizedImageClone, new Size (200, 100));
    Mat bSubmat = mOutputFrame.submat(new Rect(0, 0, 200, 100));
    mResizedImageClone.copyTo(bSubmat); 
}

What I'm trying to do is to copy a resized version of mComparatorImage into the camera frame that is referenced by mOutputFrame (mOutputFrame = rgba).

So I tried doing (3). FYI: mResizedImageClone is of type Mat and is initialized as a new Mat()

Doing (3) doesn't change the mOutputFrame.

(2) Then I tried copying the whole mComparatorImage (type Mat and same size as mOutputFrame) to mOutputFrame. This worked suprisingly.

(1) Then I thought the problem has to be something with submat, because copying the big image works, but copying a small version of it into mOutputFrame doesnt. So I tried copying a little black image into mOutputFrame. This doesn't work either, although I followed other answers here.

What could be the problem? There is no error, but the camera frame stays the same in (1) and (3)

If you need any additional info, let me know.

Isa

Answer 1

Okay, I've found it, it was a little bit tricky.

The copyTo function using submatrices works only properly if the src and the dest Mat are of the same type. Otherwise, it just does ... nothing. (It rather should complain!)

Instead of using rect, I used submat with parameters (row_start, row_end, col_start, col_end)

Also be aware that the dimensions of the submat (#cols and #rows) have to exactly match the src image that is used in copyTo.

So here is my solution for (1):

// (1) Inserting a little black rect into the camera frame:
Mat submat = mOutputFrame.submat(0, 100, 0, 100);
Mat image =  new Mat(100, 100, mOutputFrame.type(), new Scalar(0,0,0));
image.copyTo(submat);

And my solution for (3):

// (3) Resizing and inserting an arbitrary grey image into the rgba camera frame
Imgproc.resize(mComparatorImage, mResizedImageClone, new Size (200, 100));
Imgproc.cvtColor(mResizedImageClone, mResizedImageClone, Imgproc.COLOR_GRAY2RGBA);
Mat submat = mOutputFrame.submat(0, 100, 0, 200);
mResizedImageClone.copyTo(submat);