opencv C++ create Mat object from android NV21 image data buffer

Question

I have implemented an android application that starts the camera and send all the preview buffer down to native components using JNI interface. Since the preview data is in NV21 image format, I need to create a cv::Mat instance from it. I searched for it and found the below solution:

cv::Mat _yuv(height, width, CV_8UC1, (uchar *) imagebuffer);

where imagebuffer is jbyte*

However, don't get the expected image in the output image. It's all filled with some random lines etc. Does anyone know how exactly I can do it?

Answer 1

You need to convert YUV image to RGBA image.

cv::Mat _yuv(height+height/2, width, CV_8UC1, (uchar *)imagebuffer);
cv::cvtColor(_yuv, _yuv, CV_YUV2RGBA_NV21);

Usually, YUV images are 1 channel images with 1.5*height (if it were an RGB or grayscale image).

Or you could create a new Mat and pass jint array to native function and use that array to set pixels of bitmap.

jint *_out = env->GetIntArrayElements(out, 0);     

cv::Mat _yuv(height + height/2, width, CV_8UC1, (uchar*)imagebuffer);
cv::Mat _rgba(height, width, CV_8UC4, (uchar *)_out);

cv::cvtColor(_yuv, _rgba, CV_YUV2RGBA_NV21);

env->ReleaseIntArrayElements(out, _out, 0);

In Java,

bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
pixels = new int[width * height];

native_function(height, width, bytedata, pixels);

bitmap.setPixels(pixels, 0, width, 0, 0, width, height);

Answer 2

1st answer may not works as correct color image. this code is my answer.

    cv::Mat yuv(height+height/2, width, CV_8UC1,(uchar *)nv21ImageBuffer);
    cv::Mat converted(height, width, CV_8UC3);
    cv::cvtColor(yuv, converted, CV_YUV2BGR_NV21);
    cv::imwrite("anywhere/colorImage.jpg",converted);

Answer 3

If you are using native (C++) NDK camera2 API (version 24 and above). You could do using following:

#define YUV_IMG_HEIGHT      (1280)
#define YUV_IMG_WIDTH       (720)

void ndk_yuv_to_rgb_image()
{
          uint8_t *yPixel = nullptr;
          uint8_t *uPixel = nullptr;
          uint8_t *vPixel = nullptr;

          int32_t yLen = 0;
          int32_t uLen = 0;
          int32_t vLen = 0;

          cv::Mat _yuv_rgb_img, _yuv_gray_img;

          AImage_getPlaneData(yuv_image, 0, &yPixel, &yLen);
          AImage_getPlaneData(yuv_image, 1, &uPixel, &uLen);
          AImage_getPlaneData(yuv_image, 2, &vPixel, &vLen);

          uint8_t * data = new uint8_t[yLen + vLen + uLen];
          memcpy(data, yPixel, yLen);
          memcpy(data+yLen, vPixel, vLen);
          memcpy(data+yLen+vLen, uPixel, uLen);

          cv::Mat mYUV = cv::Mat(YUV_IMG_HEIGHT * 1.5, YUV_IMG_WIDTH, CV_8UC1, data);

          cv::cvtColor(mYUV, _yuv_rgb_img, CV_YUV2RGB_NV21, 3);
}

For complete NDK Camera2 APIs in C++, check my git repo