How to feed CameraX's ImageProxy to tensorflow lite model

Question

Here's my code:

imageAnalysis.setAnalyzer(ThreadManager.getInstance().cameraExecutor, new ImageAnalysis.Analyzer() {
                @SuppressLint("UnsafeOptInUsageError")
                @Override
                public void analyze(@NonNull ImageProxy image) {
                    int rotationDegrees = image.getImageInfo().getRotationDegrees();
                    // insert your code here.

                    Log.d(TAG, "starting");

                    Bitmap bmp = Bitmap.createBitmap(270,480, Bitmap.Config.ARGB_8888);
                    conv.yuvToRgb(image.getImage(), bmp);



                    ByteBuffer buffer = ByteBuffer.allocate(3*270*480*4);

                    int pixelVals[] = new int[480*270*3];

                    bmp.getPixels(pixelVals, 0, bmp.getWidth(), 0, 0, bmp.getWidth(), bmp.getHeight());

                    int itr = 0;

                    for(int i=0;i<480;++i)
                    {
                        for(int j=0;j<270;++j)
                        {
                            int x = pixelVals[itr++];

                            float red = ((x >> 16) & 0xFF) / 256.f;
                            float green = ((x >> 8) & 0xFF) / 256.f;
                            float blue = (x & 0xFF) / 256.f;

                            buffer.putFloat(red);
                            buffer.putFloat(green);
                            buffer.putFloat(blue);
                        }
                    }

                    float[][] output = new float[1][1];

                

                    tflite.run(buffer, output);

                    float out = output[0][0];

                    ContextCompat.getMainExecutor(MainActivity.this).execute(new Runnable() {
                        @Override
                        public void run() {
                            scoretv.setText(out+"");
                            Log.d(TAG, "updating");
                        }
                    });

                    image.close();
                }
            });

The problem is that tflite.run() always return the same output, no matter what the input is. I've tested my .tflite model using python script on my pc and it works correctly.

I'm not sure if buffer holds correctly converted image when run() is called, but using debugger I verified that it at least looks like correct image data, and certainly should not give the same result every time. (Buffer data changes, so output should change too)

Also, this model should output floats closer to 1.0 when object is detected, and closer to 0.0 when it isn't. If image was corruped during conversion, model should output low value (confirmed by tests on my pc).

Model always outputs 0.7001277

EDIT: This is edited version, original question is now outdated. Sorry for that, I probably should had waited a little more time before asking a question.

Answer 1

Please check out the image format of the original model first. The image format and data pattern in the Java side should be the same with the ones at the training stage in Python.

Answer 2

Because I do not see your yuvToRgb method I will give you some hints after that. So first of all put this dependency inside your build.gradle file

implementation 'org.tensorflow:tensorflow-lite-support:0.2.0'

This is for TensorFlow Lite Support library. You can find a lot of info here which is basically what I am going to write below. So with this library you can do a lot of things like convert bitmap to byteBuffer. We have to do that because the ImageProcessor cannot handle right now media.Image of YUV format that is outputed from ImageAnalysis of CameraX . This will change in a few months I hope and we can feed directly the Image to the Support library. Check below:

 //Convert image to Bitmap
    Bitmap bitmap = imageToRGB(image, image.getWidth(), image.getHeight());
    Log.v("TFLITE_w", String.valueOf(image.getWidth()));
    Log.v("TFLITE_h", String.valueOf(image.getHeight()));

    //Loads bitmap into a TensorImage.
    int imageTensorIndex = 0;
    int[] imageShape = tfLite.getInputTensor(imageTensorIndex).shape();
    DataType imageDataType = tfLite.getInputTensor(imageTensorIndex).dataType();
    //Log.v("TFLITE", String.valueOf(imageShape[0]));

    TensorImage tensorImage = new TensorImage(imageDataType);
    tensorImage.load(bitmap);

    ImageProcessor imageProcessor = new ImageProcessor.Builder()
            .add(new ResizeOp(inputSize, inputSize, ResizeOp.ResizeMethod.BILINEAR))
            //.add(new Rot90Op(numRotation))
            .build();
                    /*.add(new ResizeWithCropOrPadOp(cropSize, cropSize))
                    // To get the same inference results as lib_task_api, which is built on top of the Task
                    // Library, use ResizeMethod.BILINEAR.
                    .add(new ResizeOp(480, 640, ResizeOp.ResizeMethod.BILINEAR))
                    .add(new Rot90Op(numRotation))
                    .add(getPreprocessNormalizeOp())
                    .build();*/

    TensorImage tensorImageInput = imageProcessor.process(tensorImage);

    long endTimeForLoadImage = SystemClock.uptimeMillis();
    Trace.endSection();
    Log.v(TAG, "Time-Cost to load the image: " + (endTimeForLoadImage - startTimeForLoadImage));
    Trace.endSection(); // preprocessBitmap

    float[][] output = new float[1][1];
    tflite.run(tensorImageInput.buffer, output);

For the ImageProcessor you can see this to find out the options you want

For the method imageToRGB you can see this link if your method does'not work.

Ping me if you need more help