MediaCodec - Missing codec specific data

Question

Our project records videos using the device's microphone as audio input, and openGL scene as video - the openGL scene is a visualisation of the audio (spectrograph etc). It worked perfectly on devices until Android 11 update, but now the output videos on updated devices have no sound. The issue is not specific to any manufacturer, eg Google Pixel 3/4 and Samsung Galaxy S20 are both impacted. There are no runtime errors, or crashes and it doesn't matter what device is used to play back the file - it definitely has no sound. The audio visualisation in the video is fine, so the microphone/input is not the issue. We use MediaMuxer and MediaCodec to output the recording to an mp4 file.

Is anybody else experiencing this issue with the latest Android 11 update to devices? Does anybody have any suggestions on how we could debug?

This is how we configure our video and audio codecs:

//VIDEO OUTPUT (MP4)
public static final String VIDEO_MIME_TYPE = MediaFormat.MIMETYPE_VIDEO_AVC;
public static final int VIDEO_FRAME_RATE = 30; 
public static final int VIDEO_IFRAME_INTERVAL = 5;
public static final String AUDIO_MIME_TYPE = MediaFormat.MIMETYPE_AUDIO_AAC;Video Coding

//AUDIO INPUT (MIC)
public static final int RECORDER_SAMPLE_RATE = 44100;
public static final int RECORDER_CHANNELS = 1;
public static final int RECORDER_AUDIO_ENCODING = AudioFormat.ENCODING_PCM_16BIT;

//AUDIO OUTPUT (MP4)
public static final String ENCODER_AUDIO_FORMAT = MediaFormat.MIMETYPE_AUDIO_AAC;
public static final int ENCODER_AAC_PROFILE = MediaCodecInfo.CodecProfileLevel.AACObjectLC;
public static final int AUDIO_BIT_RATE = 128000; 
public static final int AUDIO_SAMPLE_RATE = 44100;
public static final int AUDIO_SAMPLES_PER_FRAME = 1024;
public static final int AUDIO_FRAMES_PER_BUFFER = 25;
public static final int AUDIO_BYTES_PER_ELEMENT = 2;
public static final int AUDIO_NUMBER_OF_SPECTRUM_BINS = 20;

//...

//Initialise Video Encoder
MediaFormat format = MediaFormat.createVideoFormat(Config.VIDEO_MIME_TYPE, width, height);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
format.setInteger(MediaFormat.KEY_FRAME_RATE, Config.VIDEO_FRAME_RATE);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, Config.VIDEO_IFRAME_INTERVAL);
videoEncoder = MediaCodec.createEncoderByType(Config.VIDEO_MIME_TYPE);
videoEncoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mInputSurface = videoEncoder.createInputSurface();
videoEncoder.start();

//Initialise Audio Encoder
audioEncoder = MediaCodec.createEncoderByType(Config.AUDIO_MIME_TYPE);
MediaFormat audioFormat = MediaFormat.createAudioFormat(Config.ENCODER_AUDIO_FORMAT,Config.AUDIO_SAMPLE_RATE,Config.RECORDER_CHANNELS);
audioFormat.setInteger(MediaFormat.KEY_AAC_PROFILE, Config.ENCODER_AAC_PROFILE);
audioFormat.setInteger(MediaFormat.KEY_BIT_RATE, Config.AUDIO_BIT_RATE);
audioFormat.setInteger(MediaFormat.KEY_CHANNEL_COUNT, Config.RECORDER_CHANNELS);
audioEncoder.configure(audioFormat, null, null,MediaCodec.CONFIGURE_FLAG_ENCODE);
audioEncoder.start();

//Initiate Muxer
mMuxer = new MediaMuxer(outputFile, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
mMuxer.setOrientationHint(orientationHint);

We start muxing when we receive the first frame

//...
else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
    MediaFormat videoFormat = videoEncoder.getOutputFormat();
    MediaFormat audioFormat = audioEncoder.getOutputFormat();
    videoTrackIndex = mMuxer.addTrack(videoFormat);
    audioTrackIndex = mMuxer.addTrack(audioFormat);
    mMuxer.start();
}

Inpsecting the log verbose on an impacted device, I was able to find the following lines of interest after tapping the "stop recording" button:

2021-03-08 14:52:12.658 20015-20241/com.xx.xx V/Audio:VideoEncoderCore: Sending EOS to encoder!
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: reset()
**2021-03-08 14:52:12.670 20015-20259/com.xx.xx E/MPEG4Writer: Missing codec specific data**
2021-03-08 14:52:12.670 20015-20259/com.xx.xx E/MPEG4Writer: 0 frames to dump timeStamps in Audio track 
**2021-03-08 14:52:12.670 20015-20258/com.xx.xx V/MediaWriter: Track event err/info msg:101, trackId:1, type:1000,val:-1011 **
2021-03-08 14:52:12.670 20015-20258/com.xx.xx I/MPEG4Writer: Received total/0-length (69/0) buffers and encoded 69 frames. - Video
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Video track stopping. Stop source
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Video track source stopping
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Video track source stopped
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Video track stopped. Status:0. Stop source
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Audio track stopping. Stop source
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Audio track source stopping
2021-03-08 14:52:12.670 20015-20241/com.xx.xx D/MPEG4Writer: Audio track source stopped
2021-03-08 14:52:12.670 20015-20259/com.xx.xx V/MediaWriter: Track event err/info msg:100, trackId:2, type:100,val:-1007
2021-03-08 14:52:12.670 20015-20259/com.xx.xx I/MPEG4Writer: Received total/0-length (87/0) buffers and encoded 87 frames. - Audio
2021-03-08 14:52:12.670 20015-20259/com.xx.xx I/MPEG4Writer: Audio track drift time: 0 us
2021-03-08 14:52:12.670 20015-20262/com.xx.xx I/hw-BpHwBinder: onLastStrongRef automatically unlinking death recipients
**2021-03-08 14:52:12.671 20015-20241/com.xx.xx D/MPEG4Writer: Audio track stopped. Status:-1007. Stop source **
**2021-03-08 14:52:12.671 20015-20241/com.xx.xx W/MPEG4Writer: Condition trackErr == OK failed Audio track stopped with an error **
2021-03-08 14:52:12.671 20015-20241/com.xx.xx D/MPEG4Writer: Duration from tracks range is [2050653, 2293393] us
2021-03-08 14:52:12.671 20015-20256/com.xx.xx D/MPEG4Writer: 0 chunks are written in the last batch
2021-03-08 14:52:12.671 20015-20241/com.xx.xx D/MPEG4Writer: WriterThread stopped. Status:0
2021-03-08 14:52:12.671 20015-20241/com.xx.xx I/MPEG4Writer: Adjust the moov start time from 0 us -> 0 us
2021-03-08 14:52:12.671 20015-20241/com.xx.xx I/MPEG4Writer: MOOV atom was written to the file
2021-03-08 14:52:12.671 20015-20241/com.xx.xx D/MPEG4Writer: release()

Answer 1

The following error helped me solve this problem:

2021-03-08 14:52:12.670 20015-20259/com.xx.xx E/MPEG4Writer: Missing codec specific data

More information about codec specific data and what it is.

I still don't know why this only started after the Android 11 push, but I do know the cause and how to fix it. Originally, I would pull the MediaFormat from my audio and video encoders in the video thread after receiving the first frame of video and then call MediaMuxer.start() . Up until Android 11, this approach worked fine. However, after the update, my audio format had content specific data attached which needed to be provided to the muxer. Unfortunately, this data is only available from the audio encoder after receiving the first frame of sound (not first frame of video). So, my previous approach wouldn't work as I wasn't passing audio frames until the first frame of video was received, and I was starting my muxer before receiving the codec specific data from my audio encoder.

The Solution

Wait to receive the first frame of sound and call MediaCodec.getMediaFormat() in your audio thread. Do the same for video in your video thread. Include logic to check that both the audio and video formats have been received first, and then initiate the muxer in your video thread. All my output files now have video and audio.

Answer 2

I faced the same issue while been working on some muxing stuff. The bad thing about your solution is the fact that you'll lose all videoframes which was provided to mediamuxer by camera until you'll receive first audioframe. i guess it could be better to provide some dummy audioframe at the start of encoding?