Android CameraX with custom surface

Question

In my app I am using CameraX for screen recording. I am trying to save not just camera output, but also GUI layout presented over camera surface.

I am executing some basic animations on GUI layer( showing or hiding layouts and progress bar animations).

According to my knowledge, VideoCapture is capturing camera surface. So if I create custom surface for Camera, it should be possible to capture video with GUI layout.

How to set custom surface to CameraX? I was thinking about creating custom view(surface) from PreviewView but it is not possible. (PrevieView is final class) Is it possible to create custom surface provider? How?

PreviewView

    <androidx.camera.view.PreviewView
        android:id="@+id/view_finder"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />
    ...

Start of camera

private fun startCamera() {

        val cameraProviderFuture = ProcessCameraProvider.getInstance(requireContext())

        cameraProviderFuture.addListener(Runnable {
            // Camera provider is now guaranteed to be available
            val cameraProvider = cameraProviderFuture.get()

            // Set up the preview use case to display camera preview.
            val preview = Preview.Builder().apply {
                setCameraSelector(CameraSelector.DEFAULT_FRONT_CAMERA)
            }.build()

            // Set up the capture use case to allow users to take photos.
            videoCapture = VideoCapture.Builder().apply {
                setCameraSelector(CameraSelector.DEFAULT_FRONT_CAMERA)
            }.build()

            // Choose the camera by requiring a lens facing
            val cameraSelector = CameraSelector.Builder()
                .requireLensFacing(CameraSelector.LENS_FACING_FRONT)
                .build()

            // Attach use cases to the camera with the same lifecycle owner
            val camera = cameraProvider.bindToLifecycle(
                customLifecycle as LifecycleOwner, cameraSelector, preview, videoCapture)

            // Connect the preview use case to the previewView
            preview.setSurfaceProvider(
                viewFinder.getSurfaceProvider())

        }, ContextCompat.getMainExecutor(requireContext()))
    }

Answer 1

As far as I know It is not possible to record video with an overlay in CameraX.
That's why many people use some other libs to record videos with overlay like ffmpeg. Luckily you can use MediaCodec on Android.

I believe the examples here at Grafika repository by Google would be very helpful for your case. Especially, Show + Capture Camera Activity in the readme part.

Answer 2

Unfortunately if you want to provide a custom surface you need to give up on PreviewView.

The way to do it is by creating a custom Preview.SurfaceProvider implementation and feed that provider, you will then have to provide the surface yourself as per the interface contract, for example providing a basic surface from a SurfaceHolder looks like this:

override fun onSurfaceRequested(request: SurfaceRequest) {
    surfaceHolder?.let {
        providedSurface = with(request.resolution) {
            it.setFixedSize(width, height).run { it.surface }
        }
    }

    request.provideSurface(providedSurface, executor) { result ->
        //TODO: explore the result of providing the surface
    }
}

Then you can build your custom surface provider and use it with the Preview use case as you already do.

preview.setSurfaceProvider(
            yourCustomSurfaceProvider)

As a bonus, there is MediaProjection API which records anything happening on the device screen, unfortunately it's min API 21 and has to be started through an intent.

Good luck.