MediaMuxer有時會導致黑色視頻

Question

我正在使用AndroidEncoder生成帶有AAC音頻的H264視頻，該視頻既可以通過RTMP網絡發送，也可以使用MediaMuxer本地存儲在MP4文件中。

廣播在RTMP上運行良好，但是在本地保存時，生成的視頻有時還可以，有時只是帶有聲音的黑色幀（或者只是帶有某些信息塊的第一個部分幀），有時首先播放音頻，然后播放視頻隨機FPS速度。

這是我的Muxer包裝器類和MPEG4Writer SDK類顯示的輸出：

02-28 11:57:38.521 6420-6475/com.myapp.broadcast W/AndroidMuxer: addTrack: Requested adding of track to class AndroidMuxer of type AUDIO
02-28 11:57:38.522 6420-6475/com.myapp.broadcast W/AndroidMuxer: addTrack: Adding track for audio/mp4a-latm
02-28 11:57:38.522 6420-6475/com.myapp.broadcast W/AndroidMuxer: addTrack: Track index to class AndroidMuxer of type AUDIO, track saved for future use!
02-28 11:57:38.621 6420-6470/com.myapp.broadcast W/AndroidMuxer: addTrack: Requested adding of track to class AndroidMuxer of type VIDEO
02-28 11:57:38.621 6420-6470/com.myapp.broadcast W/AndroidMuxer: addTrack: Adding track for video/avc
02-28 11:57:38.621 6420-6470/com.myapp.broadcast W/AndroidMuxer: addTrack: All tracks added, marking Local muxer as ready!
02-28 11:57:38.622 6420-6470/com.myapp.broadcast W/AndroidMuxer: addTrack: Track index to class AndroidMuxer of type VIDEO, track saved for future use!
02-28 11:57:52.050 6420-6420/com.myapp.broadcast I/AndroidMuxer: init: Added audio track widh id 0
02-28 11:57:52.050 6420-6420/com.myapp.broadcast I/AndroidMuxer: init: Added video track widh id 1
02-28 11:57:52.050 6420-6420/com.myapp.broadcast W/AndroidMuxer: init: Muxer was successfully created! Starting to record
02-28 11:57:52.056 6420-6420/com.myapp.broadcast I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3195 bytes
02-28 11:57:52.057 6420-6420/com.myapp.broadcast W/AndroidMuxer: start: Muxer started!
02-28 11:57:52.070 6420-6965/com.myapp.broadcast I/MPEG4Writer: setStartTimestampUs: 0
02-28 11:57:52.070 6420-6965/com.myapp.broadcast I/MPEG4Writer: Earliest track starting time: 0
02-28 11:57:52.083 6420-6966/com.myapp.broadcast I/MPEG4Writer: setStartTimestampUs: 0
02-28 11:58:02.617 6420-6475/com.myapp.broadcast W/AndroidMuxer: writeSampleData: All tracks finished! Calling Stop
02-28 11:58:02.617 6420-6475/com.myapp.broadcast W/AndroidMuxer: stop: Calling stop to the muxer
02-28 11:58:02.617 6420-6475/com.myapp.broadcast D/MPEG4Writer: Audio track stopping
02-28 11:58:02.617 6420-6475/com.myapp.broadcast D/MPEG4Writer: Audio track source stopping
02-28 11:58:02.617 6420-6475/com.myapp.broadcast D/MPEG4Writer: Audio track source stopped
02-28 11:58:02.617 6420-6965/com.myapp.broadcast I/MPEG4Writer: Received total/0-length (452/0) buffers and encoded 452 frames. - Audio
02-28 11:58:02.617 6420-6965/com.myapp.broadcast I/MPEG4Writer: Audio track drift time: 0 us
02-28 11:58:02.617 6420-6475/com.myapp.broadcast D/MPEG4Writer: Audio track stopped
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Video track stopping
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Video track source stopping
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Video track source stopped
02-28 11:58:02.618 6420-6966/com.myapp.broadcast I/MPEG4Writer: Received total/0-length (315/0) buffers and encoded 315 frames. - Video
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Video track stopped
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Duration from tracks range is [10143021, 10495420] us
02-28 11:58:02.618 6420-6475/com.myapp.broadcast D/MPEG4Writer: Stopping writer thread
02-28 11:58:02.620 6420-6964/com.myapp.broadcast D/MPEG4Writer: 0 chunks are written in the last batch
02-28 11:58:02.620 6420-6475/com.myapp.broadcast D/MPEG4Writer: Writer thread stopped
02-28 11:58:02.621 6420-6475/com.myapp.broadcast I/MPEG4Writer: The mp4 file will not be streamable.
02-28 11:58:02.621 6420-6475/com.myapp.broadcast W/AndroidMuxer: stop: muxer stopped!

並不是說雖然本地保存的視頻不好，但相同素材的實時流（來自相同的編碼過程）還是可以的，所以問題必須出在混音器本身（或更准確地說，是我處理它的方式） ）。

這是我的Muxer包裝器（受bigflake的mediacodec示例啟發）：

import android.media.MediaCodec;
import android.media.MediaFormat;
import android.media.MediaMuxer;
import android.util.Log;

import java.io.IOException;
import java.nio.ByteBuffer;

/**
 * @hide
 */
public class AndroidMuxer extends Muxer {
    private static final String TAG = "AndroidMuxer";
    private static final boolean VERBOSE = false;

    private MediaMuxer mMuxer;
    private boolean mStarted = false;
    private boolean mStopped = false;
    private boolean mReadyToStart;
    private MediaFormat mVideoTrackFormat;
    private MediaFormat mAudioTrackFormat;

    private long mStartTimeUs = 0;
    @Override
    public void reset() {
        Log.w(TAG, "reset: Reset is called on LOCAL MUXER!");

            mStarted = false;
            mReadyToStart = false;
            mNumTracks = 0;
            mNumTracksFinished = 0;
            Log.w(TAG, "reset: Reset worked on the Muxer");

    }

    /**
     * This will only initialize the fields, we only want the real muxer to be initialized
     * when we go live.
     * @param outputFile outputPath of the file
     * @param format Format of the muxer (will always be MPEG4 TBH)
     */
    public AndroidMuxer(String outputFile, FORMAT format) {
        super(outputFile, format);
        mStarted = false;
    }

    @Override
    public void init() {
        if(!mReadyToStart) {
            return;
        }

        if(mStarted){
            Log.i(TAG, "init: Muxer already created! Skipping!");
            return;
        }


        try {
            switch (mFormat) {
                case MPEG4:
                    mMuxer = new MediaMuxer(mOutputPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
                    int audioTrack = mMuxer.addTrack(mAudioTrackFormat);
                    Log.i(TAG, "init: Added audio track widh id " + audioTrack);
                    mAudioTrackIndex = audioTrack;

                    int videoTrack = mMuxer.addTrack(mVideoTrackFormat);
                    Log.i(TAG, "init: Added video track widh id " + videoTrack);
                    mVideoTrackIndex = videoTrack;
                    Log.w(TAG, "init: Muxer was successfully created! Starting to record");
                    start();
                    break;
                default:
                    throw new IllegalArgumentException("Unrecognized format!");
            }
        } catch (IOException e) {
            throw new RuntimeException("MediaMuxer creation failed", e);
        }
    }

    public static AndroidMuxer create(String outputFile, FORMAT format) {
        return new AndroidMuxer(outputFile, format);
    }

    @Override
    public int addTrack(MediaFormat trackFormat, AndroidEncoder.TrackType trackType) {
        Log.w(TAG, "addTrack: Requested adding of track to class " + getClass().getSimpleName() + " of type " + trackType);
        //Super method just keeps track of amount of tracks added
        super.addTrack(trackFormat, trackType);
        Log.w(TAG, "addTrack: Adding track for " + trackFormat.getString(MediaFormat.KEY_MIME));
        if (mStarted)
            throw new RuntimeException("format changed twice");

        //What we do is store the media info for later use once the muxer is created.
        //This is done to prevent file creation until the event goes live.

        if(trackType == AndroidEncoder.TrackType.VIDEO){
            mVideoTrackFormat = trackFormat;
        }
        else if(trackType == AndroidEncoder.TrackType.AUDIO){
            mAudioTrackFormat = trackFormat;
        }



        if (allTracksAdded()) {
            Log.w(TAG, "addTrack: All tracks added, marking Local muxer as ready!");
            markReadyToStart();
        }
        Log.w(TAG, "addTrack: Track index to class " + getClass().getSimpleName() + " of type " + trackType + ", track saved for future use!");
        //Return dummy
        return 1;
    }


    /**
     *
     * Marking the Muxer as ready to be started but not yet!
     *
     */
    private void markReadyToStart() {
        mReadyToStart = true;
    }



    private void start() {
        if(mStarted){
            Log.i(TAG, "start: Skipped start due to muxer already started");
        }
        mStartTimeUs = 0;
        mMuxer.start();
        Log.w(TAG, "start: Muxer started!");
        mStarted = true;
    }

    protected void stop() {
        if(mStarted) {
            Log.w(TAG, "stop: Calling stop to the muxer");
            mMuxer.stop();
            Log.w(TAG, "stop: muxer stopped!");
            mStarted = false;
            mStopped = true;
            mReadyToStart = false;
        }
    }

    @Override
    public void release() {
        super.release();
        if(mStopped) {
            mMuxer.release();
            mStopped = false;
            mStarted = false;
        }
    }

    @Override
    public boolean isStarted() {
        return mStarted;
    }

    @Override
    public void writeSampleData(MediaCodec encoder, int trackIndex, int bufferIndex, ByteBuffer encodedData, MediaCodec.BufferInfo bufferInfo) {
        if(!mStarted) {

            return;
        }

        super.writeSampleData(encoder, trackIndex, bufferIndex, encodedData, bufferInfo);
        if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
            // MediaMuxer gets the codec config info via the addTrack command
            if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
            return;
        }

        if (bufferInfo.size == 0) {
            if (VERBOSE) Log.d(TAG, "ignoring zero size buffer");
            return;
        }

        if (!mStarted) {
            Log.w(TAG, "writeSampleData called before muxer started. Ignoring packet. Track index: " + trackIndex + " tracks added: " + mNumTracks);
//            encoder.releaseOutputBuffer(bufferIndex, false);
            return;
        }

//        bufferInfo.presentationTimeUs = getNextRelativePts(bufferInfo.presentationTimeUs, trackIndex);


       if(mStartTimeUs == 0) {
              mStartTimeUs = bufferInfo.presentationTimeUs;
             }
        bufferInfo.presentationTimeUs -= mStartTimeUs;
        if(bufferInfo.presentationTimeUs < 0) {
             bufferInfo.presentationTimeUs = 0;
        }
        bufferInfo.presentationTimeUs = getSafePts(bufferInfo.presentationTimeUs, trackIndex);


        mMuxer.writeSampleData(trackIndex, encodedData, bufferInfo);

//        encoder.releaseOutputBuffer(bufferIndex, false);

        if (allTracksFinished()) {
            Log.w(TAG, "writeSampleData: All tracks finished! Calling Stop");
            stop();
        }
    }

    @Override
    public void forceStop() {
        Log.w(TAG, "forceStop: ForceStop called!");
        stop();
    }
}

請注意，RTMP多路復用器和本地多路復用器可能無法同時啟動，這就是為什么我存儲媒體信息，然后僅在用戶請求啟動多路復用器時才使用它的原因。

Answer 1

如果本地多路復用器尚未准備好並且您跳過寫入數據包，則可以跳過寫入關鍵幀。 不是關鍵幀的后來的視頻幀只有在到達下一個關鍵幀之前才能被解碼（誰知道如果文件以非關鍵幀開頭，視頻播放器是否會感到困惑？）。 如果關鍵幀之間的間隔很長，則可能在合理的時間內根本無法獲得關鍵幀。 （不過，如果您也通過RTMP進行流式處理，則可能會定期創建關鍵幀。）

我的建議是確保存儲在本地多路復用器就緒之前產生的所有數據包，並在就緒后將它們寫入本地多路復用器，或者繼續跳過數據包，直到到達下一個關鍵幀為止。