大疆手机 SDK Mavic 2 Pro - 直播 Stream 4k

Question

I'm new here in stackoverflow and I hope someone of you can help me.

I developed an application with DJI mobile SDK and I implemented a live stream. The problem is the resolution of the live stream is not 4k and I need 4k. I think the drone provides the secondary stream for the live preview. Is it possible to change the secondary stream to the primary stream which have 4k resolution? And when it is possible how can I do that? Or is it simply possible to increase the resolution of the live stream / secondary stream?

Here is my current implementation:

Initialization of surface texture element for live stream preview:

SurfaceTextureListener surfaceTextureListener = new SurfaceTextureListener(getApplicationContext());
this.videoStreamPreviewTtView.setSurfaceTextureListener(surfaceTextureListener);

This is my listener:

public class SurfaceTextureListener implements TextureView.SurfaceTextureListener {

    private final Context context;

    public SurfaceTextureListener(Context context) {
        this.context = context;
    }

    @Override
    public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
        if (DroneControl.getCodecManager() == null) {
            DroneControl.setCodecManager(new DJICodecManager(this.context, surface, width, height));
            DroneControl.getCodecManager().resetKeyFrame();
            DroneControl.getCodecManager().enabledYuvData(true);
            DroneControl.getCodecManager().setYuvDataCallback(new LiveStreamDataCallback(this.context));
        }
    }

    @Override
        public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
    }

    @Override
    public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
        if (DroneControl.getCodecManager() != null) {
            DroneControl.getCodecManager().cleanSurface();
            DroneControl.setCodecManager(null);
        }
        return false;
    }

    @Override
    public void onSurfaceTextureUpdated(SurfaceTexture surface) {
    }
}

And here is my callback function:

public class LiveStreamDataCallback implements Base, DJICodecManager.YuvDataCallback {

    private final Context context;
    private final long lastUpdate;

    public LiveStreamDataCallback(Context context) {
        this.context = context;
        this.lastUpdate = System.currentTimeMillis();
    }

    @Override
    public void onYuvDataReceived(MediaFormat format, final ByteBuffer yuvFrame, int dataSize, final int width, final int height) {
        long differenceInMillis = System.currentTimeMillis() - this.lastUpdate;

        if (differenceInMillis > SCREEN_SHOT_PERIOD && yuvFrame != null) {
            final byte[] bytes = new byte[dataSize];
            yuvFrame.get(bytes);
            newSaveYuvDataToJPEG(bytes, width, height);
        }
    }

    private void newSaveYuvDataToJPEG(byte[] yuvFrame, int width, int height) {
        if (yuvFrame.length < width * height) {
            return;
        }
        int length = width * height;

        byte[] u = new byte[width * height / 4];
        byte[] v = new byte[width * height / 4];

        for (int i = 0; i < u.length; i++) {
            u[i] = yuvFrame[length + i];
            v[i] = yuvFrame[length + u.length + i];
        }
        for (int i = 0; i < u.length; i++) {
            yuvFrame[length + 2 * i] = v[i];
            yuvFrame[length + 2 * i + 1] = u[i];
        }
        screenShot(yuvFrame, width, height);
    }

    private void screenShot(byte[] buf, int width, int height) {
        ByteArrayOutputStream bOutput = new ByteArrayOutputStream();

        YuvImage yuvImage = new YuvImage(buf,
            ImageFormat.NV21,
            width,
            height,
            null);

        yuvImage.compressToJpeg(new Rect(0,
            0,
            width,
            height), 100, bOutput);

        insertIntoDB(Base64.getEncoder().encodeToString(bOutput.toByteArray()));
    }

    private void insertIntoDB(String base64EncodedContent) {
        //only a limit of images will be saved inside the DB to avoid using too much space!
        DatabaseUtil.reduceTableContentToMaxContentIfNecessary(this.context, ScreenShotModel.ScreenShotEntry.TABLE_NAME, MAX_KEEP_COUNT_FOR_LIVE_STREAM_SCREEN_SHOTS);

        Date now = new Date();
        SimpleDateFormat dateFormat = new SimpleDateFormat(DATE_FORMAT_FOR_LOGGING, Locale.GERMANY);
        SQLiteDatabase db = DroneControl.getDbWriteAccess(this.context);

        //create a new map of values, where column names are the keys
        ContentValues values = new ContentValues();
        values.put(ScreenShotModel.ScreenShotEntry.COLUMN_NAME_DATA, base64EncodedContent);
        values.put(ScreenShotModel.ScreenShotEntry.COLUMN_NAME_CREATED, dateFormat.format(now));

        db.insert(ScreenShotModel.ScreenShotEntry.TABLE_NAME, null, values);
    }
}

Hope someone has a solution for my problem.

Thank you very much!

Best regards

Answer 1

Can't be done. 1080p is max, ocusync can't do any higher than that. The bandwidth isn't high enough, and the hardware in the drone don't support it.

I don't know anyway to do what you ask. The only thing you can do is to take an still image and download it, but it will be slow, of course, but can be used for image recognition for example. You don't say what you are going to use it for, but since you seem to be looking at frames, that may be a (slow) solution.

From DJI web: OcuSync Part of the Lightbridge family, DJI's newly developed OcuSync transmission system performs far better than Wi-Fi transmission at all transmission speeds. OcuSync also uses more effective digital compression and channel transmission technologies, allowing it to transmit HD video reliably even in environments with strong radio interference. Compared to traditional analog transmission, OcuSync can transmit video at 720p and 1080p – equivalent to a 4-10 times better quality, without a color cast, static interference, flickering or other problems associated with analog transmission. Even when using the same amount of radio transmission power, OcuSync transmits further than analog at 4.1mi (7km)

Answer 2

Thank you very much for your answer and your help!

Actually, I tried to capture 4k images first and transfer it. As you already said I try to use the frames for object detection and I need the best performance I can get. Capturing images and transferring it is very time consuming. I need approximately 1.5 seconds for capture and save the image, 3 more seconds for transfer it and, the biggest surprise for me, to read the SD card to find the newest image takes almost 11 seconds (I tried different SD cards with max class 10). In sum the whole process for one image takes 15.5 seconds, this is way too long for my purpose…

Then I thought I could do it with the live stream. The whole process with live stream takes 500 milliseconds and this is an acceptable value for my project. It is very sobering that it seems to be impossible…

Maybe there is still another opportunity to transfer images from the drone very fast?

Answer 3

I get stills from the live stream, but in another way. I use the fpv-widget (the live stream is shown on the device), and read the bmp directly from the widget. In this way you don't have to handle so much data in java.

I even do it from python, and with some quirks I got it into a opencv2 without any marshalling. Can read out 100frames/second in python, so java should be at least that fast. You might reconsider your way of doing it. Try to avoid repacking the image data. It's still only 1080, but it's very good quality I must say.

<dji.ux.widget.FPVWidget
    android:id="@+id/fpv_widget"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_centerInParent="true"
    custom:sourceCameraNameVisibility="true" />

    public Bitmap getFrameBitmap() {
        fpvWidget = findViewById(R.id.fpv_widget);
        return fpvWidget.getBitmap();
    }