ML Kit OCR in Fotoapparat returns nonsense

Question

I'm trying to do custom frame processing, to create an ML-Kit OCR app. I first used FotoApparat to create a simple camera app.

I then added a custom frame processing anonymous function in m initialization of FotoApparat.

   private fun createFotoapparat(){
        val cameraView = findViewById<CameraView>(R.id.camera_view)
        fotoapparat = Fotoapparat
            .with(this)
            .into(cameraView)
            .previewScaleType(ScaleType.CenterCrop)
            .lensPosition(back())
            .logger(loggers(logcat()))
            .cameraErrorCallback({error -> println("Recorder errors: $error")})
            .frameProcessor { frame ->
                Log.d("Frameprocessor", "Fired")
                val rotation = getRotationCompensation("0", this, baseContext)
                val BAimage = frame.image
                val metadata = FirebaseVisionImageMetadata.Builder()
                    .setWidth(480)   // 480x360 is typically sufficient for
                    .setHeight(360)  // image recognition
                    .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
                    .setRotation(rotation)
                    .build()
                var FBimage = FirebaseVisionImage.fromByteArray(BAimage, metadata)
                val detector = FirebaseVision.getInstance()
                    .onDeviceTextRecognizer
                val result = detector.processImage(FBimage)
                    .addOnSuccessListener { firebaseVisionText ->
                        Log.d("OnSuccess", "Triggered")
                        for (block in firebaseVisionText.textBlocks){
                            val blockText = block.text
                            val blockConfidence = block.confidence
                            Log.d("newframe", blockText)
                            Log.d(blockText, blockConfidence.toString())
                        }
                    }
                    .addOnFailureListener {
                        Log.e("err", "line 114", it)
                    }
            }.build()
    }

My problem is that it's returning nonsense, with a null value for the confidence. Here's some of the logcat output, when it's looking at a simple image with a small amount of typed text.

2019-03-01 14:24:56.735 16117-16117/me.paxana.myapplication D/newframe: 111
2019-03-01 14:24:56.735 16117-16117/me.paxana.myapplication D/111: null

I can post more of the code, or more of the logcat as needed, but I feel like I'm missing something major here.

Answer 1

I partially figured it out. My rotation algorithm is wrong, I have to take the picture at a 90 degree angle and then it works perfectly. This is my rotation algorithm, I'll update when I get it working.

    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    @Throws(CameraAccessException::class)
    private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        val deviceRotation = activity.windowManager.defaultDisplay.rotation
        var rotationCompensation = ORIENTATIONS.get(deviceRotation)

        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        val cameraManager = context.getSystemService(Context.CAMERA_SERVICE) as CameraManager
        val sensorOrientation = cameraManager
            .getCameraCharacteristics(cameraId)
            .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360

        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        val result: Int
        when (rotationCompensation) {
            0 -> result = FirebaseVisionImageMetadata.ROTATION_0
            90 -> result = FirebaseVisionImageMetadata.ROTATION_90
            180 -> result = FirebaseVisionImageMetadata.ROTATION_180
            270 -> result = FirebaseVisionImageMetadata.ROTATION_270
            else -> {
                result = FirebaseVisionImageMetadata.ROTATION_0
                Log.e("Err", "Bad rotation value: $rotationCompensation")
            }
        }
        return result
    }

}