[英]Problems of using MediaCodec.getOutputFormat() for an encoder in Android 4.1/4.2 devices
I'm trying to use MediaCodec to encode frames (either by camera or decoder) into a video. 我正在尝试使用MediaCodec将帧(通过摄像机或解码器)编码为视频。 When processing the encoder output by dequeueOutputBuffer(), I expect to receive the return index = MediaCodec.INFO_OUTPUT_FORMAT_CHANGED, so I can call getOutputFormat() to get the encoder output format as the input of the currently used ffmpeg muxer.
在通过dequeueOutputBuffer()处理编码器输出时,我希望收到返回索引= MediaCodec.INFO_OUTPUT_FORMAT_CHANGED,因此我可以调用getOutputFormat()来获取编码器输出格式,作为当前使用的ffmpeg多路复用器的输入。
I have tested some pad/phone devices with Android version 4.1~4.3. 我已经使用Android版本4.1〜4.3测试了一些平板/电话设备。 All of them have at least one hardware video AVC encoder and is used in the test.
它们全部具有至少一个硬件视频AVC编码器,并在测试中使用。 On the devices with version 4.3, the encoder gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED before writing the encoded data as expected, and the output format returned from getOutputFormat() can be used by the muxer correctly.
在版本为4.3的设备上,编码器在按预期方式写入编码数据之前会给出MediaCodec.INFO_OUTPUT_FORMAT_CHANGED,并且复用器可以正确使用从getOutputFormat()返回的输出格式。 On the devices with 4.2.2 or lower, the encoder never gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED while it can still output the encoded elementary stream, but the muxer cannot know the exact output format.
在4.2.2或更低版本的设备上,编码器从不提供MediaCodec.INFO_OUTPUT_FORMAT_CHANGED,尽管它仍然可以输出编码后的基本流,但是复用器无法知道确切的输出格式。
I want to ask the following questions: 我想问以下问题:
I have tried these solutions to get the output format but failed: 我尝试了以下解决方案以获取输出格式,但失败了:
Use the initial MediaFormat use to config the encoder at the beginning, like the example: 使用最初的MediaFormat用法在开始时配置编码器,例如以下示例:
m_init_encode_format = MediaFormat.createVideoFormat(m_encode_video_mime, m_frame_width, m_frame_height); int encode_bit_rate = 3000000; int encode_frame_rate = 15; int encode_iframe_interval = 2; m_init_encode_format.setInteger(MediaFormat.KEY_COLOR_FORMAT, m_encode_color_format); m_init_encode_format.setInteger(MediaFormat.KEY_BIT_RATE, encode_bit_rate); m_init_encode_format.setInteger(MediaFormat.KEY_FRAME_RATE, encode_frame_rate); m_init_encode_format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, encode_iframe_interval); m_encoder = MediaCodec.createByCodecName(m_video_encoder_codec_info.getName()); m_encoder.configure(m_init_encode_format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); // Assume m_init_encode_format is the output format of the encoder
However it fails since the output format of the encoder is still "changed" from the initial one. 但是,由于编码器的输出格式仍与初始格式“更改”,因此失败。
Please help me to realize the behavior of an encoder, and if there is any solution to query the output format if the required MediaCodec.INFO_OUTPUT_FORMAT_CHANGED is missing. 请帮助我实现编码器的行为,如果缺少所需的MediaCodec.INFO_OUTPUT_FORMAT_CHANGED,是否有任何解决方案来查询输出格式。
By comparing the output format and the codec config data, the missing fields are csd-0, csd-1, and a "what" field with value = 1869968451. (I do not understand the "what" field. It seems to be a constant and is not required. Can anyone tell me about its meaning?) 通过比较输出格式和编解码器配置数据,缺少的字段为csd-0,csd-1和值= 1869968451的“ what”字段。(我不理解“ what”字段。这似乎是一个常量,不是必需的。有人可以告诉我它的含义吗?)
If I parse the codec config data as the csd-1 field (last 8 bytes) and csd-0 field (remaining bytes), it seems that the muxer can work correctly and output a video playable on all of the testing devices. 如果我将编解码器配置数据解析为csd-1字段(最后8个字节)和csd-0字段(剩余字节),则表明多路复用器可以正常工作,并且可以在所有测试设备上输出可播放的视频。 (But I want to ask: is this 8-byte assumption correct, or there is more reliable way to parse the data?)
(但是我想问:这个8字节的假设是正确的,还是有更可靠的方法来解析数据?)
However, I got another problem that If I decode the video by Android MediaCodec again, the BufferInfo.presentationTimeUs value get by dequeueOutputBuffer() is 0 for most of the decoded frames. 但是,还有另一个问题,如果我再次通过Android MediaCodec解码视频,则对于大多数解码帧,由dequeueOutputBuffer()获得的BufferInfo.presentationTimeUs值为0。 Only the last few frames has correct time.
只有最后几帧具有正确的时间。 The sample time get by MediaExtractor.getSampleTime() is correct and exactly the value I set to the encoder/muxer, but the decoded frame time is not.
MediaExtractor.getSampleTime()获得的采样时间是正确的,并且与我为编码器/复用器设置的值完全相同,但解码的帧时间却不正确。 This issue only happen on 4.2.2 or lower device.
仅在4.2.2或更低版本的设备上会发生此问题。
It is strange that the frame time is incorrect but the video can be playback in correct speed on the device. 奇怪的是帧时间不正确,但是视频可以在设备上以正确的速度播放。 (Most of the devices with 4.2.2 or lower I've tested has only 1 Video AVC decoder.) Do I need to set other fields that may affect the presentation time?
(我测试过的大多数4.2.2或更低版本的设备只有1个Video AVC解码器。)是否需要设置其他可能影响演示时间的字段?
The behavior of MediaCodec
encoders was changed in Android 4.3 to accommodate the introduction of the MediaMuxer
class. 在Android 4.3中更改了
MediaCodec
编码器的行为,以适应MediaMuxer
类的引入。 In Android 4.3, you will always receive INFO_OUTPUT_FORMAT_CHANGED
from the encoder. 在Android 4.3中,您将始终从编码器收到
INFO_OUTPUT_FORMAT_CHANGED
。 In previous releases, you will not. 在以前的版本中,您不会。 (I've updated the relevant FAQ entry .)
(我已经更新了相关的常见问题解答条目 。)
There is no way to query the encoder for the MediaFormat
. 无法向编码器查询
MediaFormat
。
I haven't used an ffmpeg-based muxer, so I'm not sure what information it needs. 我没有使用基于ffmpeg的多路复用器,所以我不确定它需要什么信息。 If it's looking for the csd-0 / csd-1 keys, you can extract those from the
CODEC_CONFIG
packet (I think you have to parse the SPS / PPS values out and place them in the separate keys). 如果要查找csd-0 / csd-1密钥,则可以从
CODEC_CONFIG
数据包中提取这些密钥(我认为您必须解析出SPS / PPS值并将它们放在单独的密钥中)。 Examining the contents of the MediaFormat
on a 4.3 device will show you which fields you're lacking. 在4.3设备上检查
MediaFormat
的内容将向您显示您缺少哪些字段。
To init ffmpeg muxer for video correctly i use next: 要正确初始化视频的ffmpeg混合器,请使用以下命令:
int outputBufferIndex = videoCodec.dequeueOutputBuffer(bufferInfo, -1);
if (MediaCodec.BUFFER_FLAG_CODEC_CONFIG == bufferInfo.flags) {
ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
headerData = new byte[bufferInfo.size];
outputBuffer.get(headerData);
// jni call
WriteVideoHeader(headerData, headerData.length);
videoCodec.releaseOutputBuffer(outputBufferIndex, false);
}
In jni I use something like this: 在jni中,我使用如下代码:
jint Java_com_an_FileWriterEx_WriteVideoHeader(JNIEnv * env, jobject this, jbyteArray data, jint datasize)
{
jboolean isCopy;
jbyte* rawjBytes = (*env)->GetByteArrayElements(env, data, &isCopy);
WriteVideoHeaderInternal(env, m_pFormatCtx, m_pVideoStream, rawjBytes, datasize);
(*env)->ReleaseByteArrayElements(env, data, rawjBytes, 0);
return 0;
}
jint WriteVideoHeaderInternal(JNIEnv * env, AVFormatContext* pFormatCtx, AVStream* pVideoStream, jbyte* data, jint datasize)
{
jboolean bNoError = JNI_TRUE;
jbyte* pExtDataBuffer = av_malloc(datasize);
if(!pExtDataBuffer)
{
LOGI("av alloc error\n");
bNoError = JNI_FALSE;
}
if (bNoError)
{
memcpy(pExtDataBuffer, data, datasize * sizeof(jbyte));
pVideoStream->codec->extradata = pExtDataBuffer;
pVideoStream->codec->extradata_size = datasize;
}
}
For the parsing of codec config data, it is wrong that assuming the last 8 bytes are the PPS data. 对于解析编解码器配置数据,假设最后8个字节是PPS数据是错误的。 The data must be parsed according to the start code and nal_unit_type.
必须根据起始代码和nal_unit_type解析数据。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.