Android TFLite 的 Camera X ImageAnalyzer 图像格式

Question

I am attempting to analyze camera preview frames with a tflite model, using the CameraX api.

This documentation describes using the ImageAnalyzer to process incoming frames. Currently the frames are incoming as YUV, and I'm not sure how to pass YUV image data to a tflite model thats expecting an input of the shape (BATCHxWIDTHxHEIGHTx3). In the old APIs you could specify preview output formats and change it to rgb, however this page specifically says "CameraX produces images in YUV_420_888 format."

First I'm hoping someone has found a way to pass RGB to the Analyzer rather than YUV, and secondly if not, could someone suggest a way of passing a YUV image to a TFLite interpreter? The incoming image object is of the type ImageProxy and it has 3 planes, Y, U, and V.

Answer 1

AFAIK, the ImageAnalysis use case only provides images in the YUV_420_888 format (You can see it defined here ).

The official CameraX documentation provides a way to convert YUV images to RGB bitmaps, it's at the bottom of this section .

For sample code that shows how to convert a Media.Image object from YUV_420_888 format to an RGB Bitmap object, see YuvToRgbConverter.kt .

Answer 2

For anyone having this problem now. ImageAnalysis use cases now provides images in YUV_420_888 as well as RGBA_8888 which is supported in TFLite interpreter.

Usage:

val imageAnalysis = ImageAnalysis.Builder()
            .setTargetAspectRatio(AspectRatio.RATIO_16_9)
            .setTargetRotation(viewFinder.display.rotation)
            .setBackpressureStrategy(ImageAnalysis.STRATEGY_BLOCK_PRODUCER)
            .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
            .build()

imageAnalysis.setAnalyzer(executor, ImageAnalysis.Analyzer { image ->
            if (!::bitmapBuffer.isInitialized) {
                // The image rotation and RGB image buffer are initialized only once
                // the analyzer has started running
                imageRotationDegrees = image.imageInfo.rotationDegrees
                bitmapBuffer = Bitmap.createBitmap(
                    image.width, image.height, Bitmap.Config.ARGB_8888)
            }

            // Copy out RGB bits to our shared buffer
            image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer)  }
            image.close()

val imageProcessor =
        ImageProcessor.Builder()
            .add(Rot90Op(-frame.imageRotationDegrees / 90))
            .build()

    // Preprocess the image and convert it into a TensorImage for detection.
    val tensorImage = imageProcessor.process(TensorImage.fromBitmap(frame.bitmapBuffer))
    
val results = objectDetector?.detect(tensorImage)
}

Check official sample app for more details: https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android_play_services/app/src/main/java/org/tensorflow/lite/examples/objectdetection/fragments/CameraFragment.kt