如何使用SurfaceView android处理来自相机的帧

Question

I am working in project where I capture frames fps from Android camera. after each frame is converted in Bitmap to use a native library with opencv SDK and after this bitmap is showed in an ImageView. this process to consume heap memory and to affect the performance the app. I add the next tag in the manifest.xml to increase the heap memory:

<application
    android:name="app"
    android:largeHeap="true" />

With this, the app work fine when to generate an apk version in debug mode, but when to generate an apk version in release mode, the app is very slow and doesn't work fine, there is issues in the performance.

I worked with a logic using a SurfaceView component to get frames from camera, I use a SurfaceView because I need to use in background inside of the app, I want to ask you, how do I solve this problem, counting on your help.

I am using this logic to get frames:

public synchronized void onPreviewFrame(final byte[] data, final Camera camera) {
        executorService.execute(new Runnable() {
            public void run() {

      Camera.Parameters parameters = camera.getParameters();
      orientation = getOrientationDevice().getOrientation();
      orientation = normalize(orientation);

      //converter yuv to bitmap
      int width = parameters.getPreviewSize().width;
      int height = parameters.getPreviewSize().height;
      YuvImage yuv = new YuvImage(data,PreviewFormat,width, height, null);
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      yuv.compressToJpeg(new Rect(0, 0, width, height), 80, out);
      byte[] bytes = out.toByteArray();
      Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

               // rotate image 
             if (cameraId == Camera.CameraInfo.CAMERA_FACING_FRONT){
                    matrix.setScale(-1, 1);
                    matrix.postTranslate(bitmap.getWidth(), 0);
                    matrix.postRotate(rotation + 90);
                } else {
                    matrix.postRotate(rotation + 90);
                }                        

               // process bitmap from method jni
                ....

               // to pass bitmap a ImageView to show
               mCameraFrame.onFrame(bitmap.copy(bitmap.getConfig(), false));


            if (bitmap != null && !bitmap.isRecycled()) {
                bitmap.recycle();
                bitmap = null;
            }


            camera.addCallbackBuffer(data);

            return;

            });

        }

    };

in the UI the callback receive the bitmap to show in Imageview

@Override
    public void onFrame(final Bitmap mBitmap) {

        this.bitmapWeakReference = new WeakReference<Bitmap>(mBitmap);

        if (imageViewReference != null && bitmapWeakReference.get() != null) {
            final ImageView imageView = imageViewReference.get();
            if (imageView != null) {

                imageView.post(new Runnable() {
                    @Override
                    public void run() {
                        imageView.setImageBitmap(bitmapWeakReference.get());
                    }
                });

            }
        }

Answer 1

Working with camera API at high frame rate requires to push camera callbacks off the main UI thread, you will need a separate Handler thread for that. Get rid of all unnecessary processing in your onPreviewFrame() callback. You can pre-calculate the camera parameters. Avoid new , even for Rect: garbage collection may be killing performance.

You can feed a YUV frame via JNI to native OpenCV-based JNI and all transformations and conversions may be performed much faster.

If you don't use OpenCV for every bitmap, you can convert YUV to RGB without going through JPEG. Rotation by 90° can be performed in one step with such conversion.

If you are running on a multi-core device, you can start the executorService with newFixedThreadPool(), but then release of camera buffers (calls to camera.addCallbackBuffer() ) may require delicate management, especially if you want the frame rate to be uniform.

If you simply want to show two copies of the preview stream, you can use OpenGL, and CPU will not be involved in rotation and YUV to RGB conversion at all. You can even apply some advanced image processing to texture displayed via OpenGL.