How to get raw preview data from Camera object at least 15 frames per second in Android?

Question

I need to obtain raw preview data from Camera object at least 15 frame per second , but I can only get a frame in 110 milliseconds which means I can get only 9 frames per second. I brief my code below.

Camera mCamera = Camera.open();
Camera.Parameters parameters = mCamera.getParameters();
parameters.setPreviewFrameRate(30);
parameters.setPreviewFpsRange(15000,30000);
mCamera.setParameters(parameters);
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
//dataBufferSize stands for the byte size for a picture frame
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
mCamera.addCallbackBuffer(new byte[dataBufferSize]);
mCamera.setPreviewDisplay(videoCaptureViewHolder);
//videoCaptureViewHolder is a SurfaceHolder object
mCamera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {
  private long timestamp=0;
  public synchronized void onPreviewFrame(byte[] data, Camera camera) {
    Log.v("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
    timestamp=System.currentTimeMillis();
    //do picture data process
    camera.addCallbackBuffer(data);
    return;
  }
}
mCamera.startPreview();

In the briefed code above, dataBufferSize and videoCaptureViewHolder is defined and calculated or assigned in other statements.

I run my code, I can see preview on the screen and I get the log below:

...
V/CameraTest( 5396): Time Gap = 105
V/CameraTest( 5396): Time Gap = 112
V/CameraTest( 5396): Time Gap = 113
V/CameraTest( 5396): Time Gap = 115
V/CameraTest( 5396): Time Gap = 116
V/CameraTest( 5396): Time Gap = 113
V/CameraTest( 5396): Time Gap = 115
...

This means onPreviewFrame(byte[] data, Camera camera) is called every 110 milliseconds so I can get no more than 9 frames per second. And no matter what preview frame rate I set by issue setPreviewFrameRate() and what preview Fps range I set by issue setPreviewFpsRange() , the log is the same.

Would some one give me some help on this problem? I need to obtain raw preview data from Camera object at least 15 frames per second. Thank you in advance.

I put my entire code below.

CameraTest.java

package test.cameratest;

import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import android.app.Activity;
import android.graphics.ImageFormat;
import android.hardware.Camera;
import android.hardware.Camera.ErrorCallback;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.Size;
import android.os.Bundle;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceView;
import android.view.SurfaceHolder.Callback;

public class CameraTestActivity extends Activity {
    SurfaceView mVideoCaptureView;
    Camera mCamera;
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        mVideoCaptureView = (SurfaceView) findViewById(R.id.video_capture_surface);
        SurfaceHolder videoCaptureViewHolder = mVideoCaptureView.getHolder();
        videoCaptureViewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
        videoCaptureViewHolder.addCallback(new Callback() {
            public void surfaceDestroyed(SurfaceHolder holder) {
            }

            public void surfaceCreated(SurfaceHolder holder) {
                startVideo();
            }

            public void surfaceChanged(SurfaceHolder holder, int format,
                    int width, int height) {
            }
        });
    }
    private void startVideo() {
        SurfaceHolder videoCaptureViewHolder = null;
        try {
            mCamera = Camera.open();
        } catch (RuntimeException e) {
            Log.e("CameraTest", "Camera Open filed");
            return;
        }
        mCamera.setErrorCallback(new ErrorCallback() {
            public void onError(int error, Camera camera) {
            }
        }); 
        Camera.Parameters parameters = mCamera.getParameters();
        parameters.setPreviewFrameRate(30);
        parameters.setPreviewFpsRange(15000,30000);
        List<int[]> supportedPreviewFps=parameters.getSupportedPreviewFpsRange();
        Iterator<int[]> supportedPreviewFpsIterator=supportedPreviewFps.iterator();
        while(supportedPreviewFpsIterator.hasNext()){
            int[] tmpRate=supportedPreviewFpsIterator.next();
            StringBuffer sb=new StringBuffer();
            sb.append("supportedPreviewRate: ");
            for(int i=tmpRate.length,j=0;j<i;j++){
                sb.append(tmpRate[j]+", ");
            }
            Log.v("CameraTest",sb.toString());
        }

        List<Size> supportedPreviewSizes=parameters.getSupportedPreviewSizes();
        Iterator<Size> supportedPreviewSizesIterator=supportedPreviewSizes.iterator();
        while(supportedPreviewSizesIterator.hasNext()){
            Size tmpSize=supportedPreviewSizesIterator.next();
            Log.v("CameraTest","supportedPreviewSize.width = "+tmpSize.width+"supportedPreviewSize.height = "+tmpSize.height);
        }

        mCamera.setParameters(parameters);
        if (null != mVideoCaptureView)
            videoCaptureViewHolder = mVideoCaptureView.getHolder();
        try {
            mCamera.setPreviewDisplay(videoCaptureViewHolder);
        } catch (Throwable t) {
        }
        Log.v("CameraTest","Camera PreviewFrameRate = "+mCamera.getParameters().getPreviewFrameRate());
        Size previewSize=mCamera.getParameters().getPreviewSize();
        int dataBufferSize=(int)(previewSize.height*previewSize.width*
                               (ImageFormat.getBitsPerPixel(mCamera.getParameters().getPreviewFormat())/8.0));
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.addCallbackBuffer(new byte[dataBufferSize]);
        mCamera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {
            private long timestamp=0;
            public synchronized void onPreviewFrame(byte[] data, Camera camera) {
                Log.v("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
                timestamp=System.currentTimeMillis();
                try{
                    camera.addCallbackBuffer(data);
                }catch (Exception e) {
                    Log.e("CameraTest", "addCallbackBuffer error");
                    return;
                }
                return;
            }
        });
        try {
            mCamera.startPreview();
        } catch (Throwable e) {
            mCamera.release();
            mCamera = null;
            return;
        }
    }
    private void stopVideo() {
        if(null==mCamera)
            return;
        try {
            mCamera.stopPreview();
            mCamera.setPreviewDisplay(null);
            mCamera.setPreviewCallbackWithBuffer(null);
            mCamera.release();
        } catch (IOException e) {
            e.printStackTrace();
            return;
        }
        mCamera = null;
    }
    public void finish(){
        stopVideo();
        super.finish();
    };
}

AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="test.cameratest"
      android:versionCode="1"
      android:versionName="1.0">
    <uses-sdk android:minSdkVersion="9" android:targetSdkVersion="10" android:maxSdkVersion="10"/>
    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.VIBRATE" />
    <uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
    <uses-permission android:name="android.permission.WAKE_LOCK" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.MOUNT_UNMOUNT_FILESYSTEMS"/>
    <uses-permission android:name="android.permission.READ_CONTACTS"/>
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
    <uses-permission android:name="android.permission.PROCESS_OUTGOING_CALLS"/>
    <uses-permission android:name="android.permission.CALL_PHONE"/>
    <uses-permission android:name="android.permission.BOOT_COMPLETED"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_SETTINGS" />    
    <application android:icon="@drawable/icon" android:label="@string/app_name">
        <activity android:name=".CameraTestActivity"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>
</manifest>

Answer 1

I'm affraid, you can not. Preview framerate setting is hint for camera appplication ( which runs in separate process) - and it is free to accept or silently ignore it. It is also not related with preview frame retrieval

When you request preview frame, you just say external application that you would like to have it. Buffer for it is allocated in camera application and then passed to your activity via mmaped memory segment - this takes time.

You may get desired performance on some devices, but not necessarily on one you are playing with.

If you need defined frame rate, you will have to capture video and then parse / decompress resulting binary stream.

Answer 2

Me experience with the camera stuff has been fiddly and hardware dependent. So try running it on other hardware sometime if you can.

Also might be worth trying some more camera settings.

• Checkout setRecordingHint (boolean hint) .
• Could even try FOCUS_MODE .

Thanks for including a code sample btw.

Answer 3

This should not be a problem. My androangelo (it's in the market) app get's up to 30 frames per second (at least I implemented a rate-brake to slow it down).

Please check carefully, whether Your log is filled with garbage-collector statements. This is the case if too few buffers are added. This had be the trick for me. At least I came up to add 20! buffers to the camera.

Then the processing of each frame should take place in a separate thread. While an image is in the thread for processing, the callback should skip the current frame.

Answer 4

One thing that seems to increase the fluidity of the preview, if not the actual FPS necessarily, is setting the previewFormat to YV12 if supported. It's fewer bytes to copy, 16-byte aligned, and possibly optimized in other ways:

    // PREVIEW FORMATS
    List<Integer> supportedPreviewFormats = parameters.getSupportedPreviewFormats();
    Iterator<Integer> supportedPreviewFormatsIterator = supportedPreviewFormats.iterator();
    while(supportedPreviewFormatsIterator.hasNext()){
        Integer previewFormat =supportedPreviewFormatsIterator.next();
        // 16 ~ NV16 ~ YCbCr
        // 17 ~ NV21 ~ YCbCr ~ DEFAULT
        // 4  ~ RGB_565
        // 256~ JPEG
        // 20 ~ YUY2 ~ YcbCr ...
        // 842094169 ~ YV12 ~ 4:2:0 YCrCb comprised of WXH Y plane, W/2xH/2 Cr & Cb. see documentation
        Log.v("CameraTest","Supported preview format:"+previewFormat);
        if (previewFormat == ImageFormat.YV12) {
            parameters.setPreviewFormat(previewFormat);
            Log.v("CameraTest","SETTING FANCY YV12 FORMAT");                
        }
    }

http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12 describes the format. This plus a few spare buffers gives me "Time Gaps" of as low as 80...which is still not "good enough", but ... better? (actually I've got one at 69...but really, they're more around 90 on average). Not sure how much logging is slowing things down?

Setting the previewSize to 320x240 (versus 1280x720) gets things down to the 50-70msec range...so maybe that's what you need to do? Admittedly, that little data may be a lot less useful.

// all tested on Nexus4

Answer 5

In my understanding, Android does not allow user to a set a fixed framerate, nor guarantee the value of fps that you specify will be respected is due to the frame exposure time which is set by the camera hardware or firmware. The frame rate you observe may be a function of lighting conditions. For example, a certain phone may give you a 30fps preview rate in a day light but only 7 fps if you are filming in a low light condition.

Answer 6

I usually declare a global boolean lockCameraUse. The the callback function usually looks like this.

  public void onPreviewFrame(byte[] data, Camera camera) {
    if (lockCameraUse) {
      camera.addCallbackBuffer(data);
      return;
    }
    lockCameraUse = true;
    // processinng data

    // done processing data
    camera.addCallbackBuffer(data);
    lockCameraUse = false;
    return;
  }