Project Tango原始数据（IMU和RGBD）

Question

I'm a beginner in android and with the Tango (I just wanted to say it to prevent possible mistakes) and I'm using the Project Tango Tablet (Yellowstone). I want to do slam with it, that's why I want to get the raw data.

Currently I know how to get poseData and cloud of point. I read that it was not possible to get IMU and RGBD. But instead of RGBD the RGB picture wich is display by some sample could be enought, but I don't know get it. I think that it will probably works with "onFrameAvailable(int i)" but I don't really know how to get the RGB picture.

For the IMU data I saw that there is an EVENT_IMU with "onTangoEvent(TangoEvent tangoEvent)". But it never happens/is never handled when it's supposed to be a high frequency event right ? However I handle the EVENT_FEATURE_TRACKING (= 5) and EVENT_FISHEYE_CAMERA (= 2) so "onTangoEvent(TangoEvent tangoEvent)" works. So my questions are the following :

1. How can I get the RGB picture ? (in matrix, buffer or anything else)
2. Why the EVENT_IMU is never handled ?
3. Is there a way to get the IMU data or something equivalent ?

Thanks for your answers

Answer 1

I forgot to say it, but I solved my problem. The Tango's API doesn't allow to get the raw data (IMU and RGBD). But you can get the RGB with the c api via the TangoImageBuffer. You will get an MV21 picture. You will be able to convert each pixel in "RGB single pixel" and then convert this pixel in RGBA data.

For the IMU, just forget the Tango's API and use the android API. You can get it easily and then transfer it to a server. I my case I do this by using ros publisher. Thanks to this I'm able to get something like 500 poses/sec, 35 point cloud/sec and 900 IMU/sec. If you are interested by using ros you can check this ROS Rviz visualization of Tango Pose data .

----- EDIT -------- Here the code from my git (see next question in comment) as I'll delete it :

import android.Manifest;
import android.app.Activity;
import android.content.Context;

import android.content.pm.PackageManager;
import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorEventListener;
import android.hardware.SensorManager;

import android.location.Location;
import android.location.LocationListener;
import android.location.LocationManager;

import android.os.Bundle;
import android.support.v4.app.ActivityCompat;

public class IMU implements SensorEventListener {
    private long timestampAccelerometerSensor, timestampLinearAccelerationSensor, timestampRotationSensor;
    private long timestampGravitySensor, timestampGyroscopeSensor, timestampMagneticFieldSensor;
    private String  tAccelerometerSensor, tLinearAccelerationSensor, tRotationSensor;
    private String tGravitySensor, tGyroscopeSensor, tMagneticFieldSensor, tGPS;

    private long countAccelerometerSensor, countLinearAccelerationSensor, countRotationSensor;
    private long countGravitySensor, countGyroscopeSensor, countMagneticFieldSensor;

    //for GPS
    private LocationManager locationManager;
    private LocationListener locationListener;
    private double longitude, latitude;

    //for IMU
    private SensorManager mSensorManager;
    private Sensor accelerometerSensor;
    private Sensor linearAccelerationSensor;
    private Sensor rotationSensor;
    private Sensor gravitySensor;
    private Sensor gyroscopeSensor;

    private Sensor magneticFieldSensor;

    float accelerationForce[] = new float[3];//Including gravity, Acceleration force along x,y,z axis in m/s²
    float linearAccelerationForce[] = new float[3];//Excluding gravity, Acceleration force along x,y,z axis in m/s²
    float gravity[] = new float[3];//Force of gravity along x,y,z axis in m/s²
    float gyroscope[] = new float[3];//Rate of rotation around x,y,z axis in rad/s
    float rotation[] = new float[4]; //Rotation vector component along the x,y,z axis and Scalar component of the rotation vector
    float magneticField[] = new float[3];//Geomagnetic field strength along x,y,z axis uT


public void create(){
        mSensorManager = (SensorManager) mainActivity.getSystemService(Context.SENSOR_SERVICE);

        locationManager = (LocationManager) mainActivity.getSystemService(Context.LOCATION_SERVICE);
        locationListener = new LocationListener() {
            @Override
            public void onLocationChanged(Location location) {
                //gpsView.setText("Longitude: " + location.getLongitude() + "\nLatitude: " + location.getLatitude());
                //Log.i("GPS","Longitude: " + location.getLongitude() + "\nLatitude: " + location.getLatitude());
                tGPS = TangoJNINative.getCurrentTimeMillisString();
                longitude = location.getLongitude();
                latitude = location.getLatitude();
            }

            @Override
            public void onStatusChanged(String provider, int status, Bundle extras) {

            }

            @Override
            public void onProviderEnabled(String provider) {
                //gpsStatusView.setText("GPS ---> On");
            }

            @Override
            public void onProviderDisabled(String provider) {
                //gpsStatusView.setText("GPS ---> Off");
            }
        };

        if (ActivityCompat.checkSelfPermission(mainActivity, Manifest.permission.ACCESS_FINE_LOCATION) != PackageManager.PERMISSION_GRANTED && ActivityCompat.checkSelfPermission(mainActivity, Manifest.permission.ACCESS_COARSE_LOCATION) != PackageManager.PERMISSION_GRANTED) {
            // TODO: Consider calling
            //    ActivityCompat#requestPermissions
            // here to request the missing permissions, and then overriding
            //   public void onRequestPermissionsResult(int requestCode, String[] permissions,
            //                                          int[] grantResults)
            // to handle the case where the user grants the permission. See the documentation
            // for ActivityCompat#requestPermissions for more details.
            return;
        }
        locationManager.requestLocationUpdates(LocationManager.NETWORK_PROVIDER, 0, 0, locationListener);

        String locationProvider = LocationManager.NETWORK_PROVIDER;
        Location firstLocation = locationManager.getLastKnownLocation(locationProvider);
        longitude = firstLocation.getLongitude();
        latitude = firstLocation.getLatitude();

        setSensors();
        setSensorsListeners();
    }

public void pause(){
        mSensorManager.unregisterListener(this);
    }

    public void resume(){
        setSensorsListeners();
    }

    private void setSensors(){
        accelerometerSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
        linearAccelerationSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_LINEAR_ACCELERATION);
        rotationSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_ROTATION_VECTOR);
        gravitySensor = mSensorManager.getDefaultSensor(Sensor.TYPE_GRAVITY);
        gyroscopeSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_GYROSCOPE);

        magneticFieldSensor = mSensorManager.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
    }

    private void setSensorsListeners(){
        mSensorManager.registerListener(this, accelerometerSensor , SensorManager.SENSOR_DELAY_NORMAL);
        mSensorManager.registerListener(this, linearAccelerationSensor , SensorManager.SENSOR_DELAY_NORMAL);
        mSensorManager.registerListener(this, rotationSensor , SensorManager.SENSOR_DELAY_NORMAL);
        mSensorManager.registerListener(this, gravitySensor , SensorManager.SENSOR_DELAY_NORMAL);
        mSensorManager.registerListener(this, gyroscopeSensor , SensorManager.SENSOR_DELAY_NORMAL);

        mSensorManager.registerListener(this, magneticFieldSensor , SensorManager.SENSOR_DELAY_NORMAL);
    }

    @Override
    public void onSensorChanged(SensorEvent event) {
        getIMU(event);
    }

    @Override
    public void onAccuracyChanged(Sensor sensor, int accuracy) {

    }

    private void getIMU(SensorEvent e){
        switch(e.sensor.getType()){
            case Sensor.TYPE_ACCELEROMETER:
                accelerationForce[0] = e.values[0];
                accelerationForce[1] = e.values[1];
                accelerationForce[2] = e.values[2];
                countAccelerometerSensor++;
                timestampAccelerometerSensor = e.timestamp;
                tAccelerometerSensor = TangoJNINative.getCurrentTimeMillisString();
                break;
            case Sensor.TYPE_LINEAR_ACCELERATION:
                linearAccelerationForce[0] = e.values[0];
                linearAccelerationForce[1] = e.values[1];
                linearAccelerationForce[2] = e.values[2];
                countLinearAccelerationSensor++;
                timestampLinearAccelerationSensor = e.timestamp;
                tLinearAccelerationSensor = TangoJNINative.getCurrentTimeMillisString();
                break;
            case Sensor.TYPE_ROTATION_VECTOR:
                rotation[0] = e.values[0];
                rotation[1] = e.values[1];
                rotation[2] = e.values[2];
                rotation[3] = e.values[3];
                countRotationSensor++;
                timestampRotationSensor = e.timestamp;
                tRotationSensor = TangoJNINative.getCurrentTimeMillisString();
                break;
            case Sensor.TYPE_GRAVITY:
                gravity[0] = e.values[0];
                gravity[1] = e.values[1];
                gravity[2] = e.values[2];
                countGravitySensor++;
                timestampGravitySensor = e.timestamp;
                tGravitySensor = TangoJNINative.getCurrentTimeMillisString();
                break;
            case Sensor.TYPE_GYROSCOPE:
                gyroscope[0] = e.values[0];
                gyroscope[1] = e.values[1];
                gyroscope[2] = e.values[2];
                countGyroscopeSensor++;
                timestampGyroscopeSensor= e.timestamp;
                tGyroscopeSensor = TangoJNINative.getCurrentTimeMillisString();
                break;
            case Sensor.TYPE_MAGNETIC_FIELD:
                magneticField[0] = e.values[0];
                magneticField[1] = e.values[1];
                magneticField[2] = e.values[2];
                countMagneticFieldSensor++;
                timestampMagneticFieldSensor = e.timestamp;
                tMagneticFieldSensor = TangoJNINative.getCurrentTimeMillisString();
                break;
        }
    }
}

Answer 2

maybe it's too late to ask you. But is there any chance that you can give me your code to get IMU measurements and RGB-video from Project-Tango. I'm dealing with this problem now. Tks a lot!