从 TensorFlow Lite Model 的图像中获取 ByteBuffer

Question

I am creating an android app to run on Google Glass Enterprise Edition 2 that does Real-time Face Recognition. I am using Camera X as my Camera API and TensorFlow Lite (TFLite) as my classification model. However, the TFLite model input requires ByteBuffer which I am unable to convert into from the image retrieved from CameraX.

How do I get my Image from CameraX into ByteBuffer class for my TFLite Model?

Camera X Image Analysis: Reference

            val imageAnalysis = ImageAnalysis.Builder()
                    .setTargetResolution(Size(640, 360))
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                    .build()

            imageAnalysis.setAnalyzer(AsyncTask.THREAD_POOL_EXECUTOR, ImageAnalysis.Analyzer { imageProxy ->
                val rotationDegrees = imageProxy.imageInfo.rotationDegrees
                val mediaImage = imageProxy.image

                if (mediaImage != null) {
                    val image = InputImage.fromMediaImage(mediaImage, rotationDegrees)

                    /* Classify the Image using TensorFlow Lite Model */

                }

            })

TensorFlow Model Sample Code

val model = FaceRecognitionModel.newInstance(context)

// Creates inputs for reference.
val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
inputFeature0.loadBuffer(byteBuffer)

// Runs model inference and gets result.
val outputs = model.process(inputFeature0)
val outputFeature0 = outputs.outputFeature0AsTensorBuffer

// Releases model resources if no longer used.
model.close()

Answer 1

Try using the TensorImage class from the TensorFlow Lite Support Library .

Roughly, you can follow these steps.

1. Convert the Image object into Bitmap . There should be other Stackoverflow questions on how to do this (eg, this answer )
2. Create a TensorImage object from the Bitmap object using TensorImage.fromBitmap() factory.
3. Call getBuffer() method on the TensorImage object to get the underlying ByteBuffer .

You might also want to do some image pre-processing, in case the image from CameraX doesn't exactly match the format expected by the model. For this, you can explore the ImageProcessor utility .