android camera2 api - transfering preview frames via socket to server

Question

I have gone through the android docs as well as the camera2basic repo and have been able to display camera preview frames on a TextureView. I have two buttons: one that can record the video, the other can capture still images and store it in the device's memory. What I wanna achieve is something different, I want to have all the camera frames being sent to the server in a sequence via sockets in real-time. I have broken down task into smaller chuncks and am presently trying to capture all the camera frames as jpeg images. This is the CameraCaptureSession.CaptureCallback code whereby I am calling savePreviewShot() which actually replicates the original captureStillPicture() function of Camera2basic repo:

private CameraCaptureSession.CaptureCallback mCaptureCallback
        = new CameraCaptureSession.CaptureCallback() {

    private void process(CaptureResult result) {
        switch (mState) {
            case STATE_PREVIEW: {
                // We have nothing to do when the camera preview is working normally.
                    savePreviewShot();
                break;
            }
            case STATE_WAITING_LOCK: {
                Integer afState = result.get(CaptureResult.CONTROL_AF_STATE);
                if (afState == null) {
                    captureStillPicture();
                } else if (CaptureResult.CONTROL_AF_STATE_FOCUSED_LOCKED == afState ||
                        CaptureResult.CONTROL_AF_STATE_NOT_FOCUSED_LOCKED == afState) {
                    // CONTROL_AE_STATE can be null on some devices
                    Integer aeState = result.get(CaptureResult.CONTROL_AE_STATE);
                    if (aeState == null ||
                            aeState == CaptureResult.CONTROL_AE_STATE_CONVERGED) {
                        mState = STATE_PICTURE_TAKEN;
                        captureStillPicture();
                    } else {
                        runPrecaptureSequence();
                    }
                }
                break;
            }
            case STATE_WAITING_PRECAPTURE: {
                // CONTROL_AE_STATE can be null on some devices
                Integer aeState = result.get(CaptureResult.CONTROL_AE_STATE);
                if (aeState == null ||
                        aeState == CaptureResult.CONTROL_AE_STATE_PRECAPTURE ||
                        aeState == CaptureRequest.CONTROL_AE_STATE_FLASH_REQUIRED) {
                    mState = STATE_WAITING_NON_PRECAPTURE;
                }
                break;
            }
            case STATE_WAITING_NON_PRECAPTURE: {
                // CONTROL_AE_STATE can be null on some devices
                Integer aeState = result.get(CaptureResult.CONTROL_AE_STATE);
                if (aeState == null || aeState != CaptureResult.CONTROL_AE_STATE_PRECAPTURE) {
                    mState = STATE_PICTURE_TAKEN;
                    captureStillPicture();
                }
                break;
            }
        }
    }

This is the savePreviewShot() function:

private void savePreviewShot(){
    try {
        final Activity activity = getActivity();
        if (null == activity || null == mCameraDevice) {
            return;
        }
        // This is the CaptureRequest.Builder that we use to take a picture.
        final CaptureRequest.Builder captureBuilder =
                mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
        captureBuilder.addTarget(mImageReader.getSurface());

        // Orientation
        int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
        captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, ORIENTATIONS.get(rotation));

        CameraCaptureSession.CaptureCallback CaptureCallback
                = new CameraCaptureSession.CaptureCallback() {

            @Override
            public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request,
                                           TotalCaptureResult result) {
                SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd_HH:mm:ss:SSS");
                Date resultdate = new Date(System.currentTimeMillis());
                String mFileName = sdf.format(resultdate);
                mFile = new File(getActivity().getExternalFilesDir(null), "pic "+mFileName+" preview.jpg");

                Log.i("Saved file", ""+mFile.toString());
                unlockFocus();
            }
        };

        mCaptureSession.stopRepeating();
        mCaptureSession.capture(captureBuilder.build(), CaptureCallback, null);
    } catch (Exception e) {
        e.printStackTrace();
    }
};

However, as you might have figured it out already, such a call causes tremendous lag in displaying the preview on the map and moreover, I am not getting the frames saved as well. Hope you have got the aim I have and the problem I am facing. I have already gone through lots of stuffs on SO and Google but none of them did help.

Answer 1

Have the same exact problem and I was wondering if you got a fix. What I did was set an ImageReader;

private ImageReader videoReader;
private final ImageReader.OnImageAvailableListener onVideoFrameAvailableListener

Loaded it when i openned the camera

videoReader = ImageReader.newInstance(videoImageSize.getWidth(), videoImageSize.getHeight(), ImageFormat.JPEG, 2);
videoReader.setOnImageAvailableListener(onVideoFrameAvailableListener, backgroundHandler);

Added the Surface to the captureSession and processed the image in the listener using frame = reader.acquireLatestImage(); But yeah, lags a lot the preview and don't know what the solution may be.