如何在Android MediaCodec中播放原始NAL单元

Question

I initially tried How to play raw NAL units in Andoid exoplayer? but I noticed I'm gonna have to do things in low level.

I've found this simple MediaCodec example . As you can see, it's a thread that plays a file on a surface passed to it.

Notice the lines

mExtractor = new MediaExtractor();
mExtractor.setDataSource(filePath);

It looks like that I should create my own MediaExtractor which, instead of extracting the video units from a file, it'll use the h264 NAL units from a buffer I'll provide.

I can then call mExtractor.setDataSource(MediaDataSource dataSource) , see MediaDataSource

It has readAt(long position, byte[] buffer, int offset, int size)

This is where it reads the NAL units. However, how should I pass them? I have no information on the structure of the buffer that needs to be read.

Should I pass a byte[] buffer with the NAL units in it, and if so, in which format? What is the offset for? If it's a buffer, shouldn't I just erase the lines that were read and thus have no offset or size?

By the way, the h264 NAL units are streaming ones, they come from RTP packets, not files. I'm gonna get them through C++ and store them on a buffer an try to pass to the mediaExtractor.

UPDATE:

I've been reading a lot about MediaCodec and I think I understand it better. According to https://developer.android.com/reference/android/media/MediaCodec , everything relies on something of this type:

 MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
   @Override
   void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
     ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
     // fill inputBuffer with valid data
     …
     codec.queueInputBuffer(inputBufferId, …);
   }

   @Override
   void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is equivalent to mOutputFormat
     // outputBuffer is ready to be processed or rendered.
     …
     codec.releaseOutputBuffer(outputBufferId, …);
   }

   @Override
   void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     mOutputFormat = format; // option B
   }

   @Override
   void onError(…) {
     …
   }
 });
 codec.configure(format, …);
 mOutputFormat = codec.getOutputFormat(); // option B
 codec.start();
 // wait for processing to complete
 codec.stop();
 codec.release();

As you can see, I can pass input buffers and get decoded output buffers. The exact byte formats are still a mystery, but I think that's how it works. Also according to the same article, the usage of ByteBuffer s is slow, and Surface s are preferred. They consume the output buffers automatically. Although there's no tutorial on how to do it, there's a section in the article that says it's almost identical, so I guess I just need to add the additional lines

 codec.setInputSurface(Surface inputSurface) 
 codec.setOutputSurface(Surface outputSurface) 

Where inputSurface and outputSurface are Surface s which I pass to a MediaPlayer which I use (how) to display the video in an activity. And the output buffers will simply not come on onOutputBufferAvailable (because the surface consumes them first), neither onInputBufferAvailable .

So the questions now are: how exactly do I construct a Surface which contains the video buffer, and how do I display a MediaPlayer into an activity

For output I can simply create a Surface and pass to a MediaPlayer and MediaCodec , but what about input? Do I need ByteBuffer for the input anyways, and Surface would just be for using other outputs as inputs?

Answer 1

you first need to remove the NAL units , and feed the raw H264 bytes into this method, how ever in your case your reading from the file , so no need to remove any thing since your not using packets , just feed the data bytes to this method:

 rivate void initDecoder(){
    try {
        writeHeader = true;
        if(mDecodeMediaCodec != null){
            try{
                mDecodeMediaCodec.stop();
            }catch (Exception e){}
            try{
                mDecodeMediaCodec.release();
            }catch (Exception e){}
        }
        mDecodeMediaCodec = MediaCodec.createDecoderByType(MIME_TYPE);
       //MIME_TYPE = video/avc    in your case
        mDecodeMediaCodec.configure(format,mSurfaceView.getHolder().getSurface(),
                null,
                0);
        mDecodeMediaCodec.start();
        mDecodeInputBuffers = mDecodeMediaCodec.getInputBuffers();
    } catch (IOException e) {
        e.printStackTrace();
        mLatch.trigger();
    }
}


   private void decode(byte[] data){


            try {

                MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
                int inputBufferIndex = mDecodeMediaCodec.dequeueInputBuffer(1000);//
                if (inputBufferIndex >= 0) {
                    ByteBuffer buffer = mDecodeInputBuffers[inputBufferIndex];
                    buffer.clear();
                    buffer.put(data);
                    mDecodeMediaCodec.queueInputBuffer(inputBufferIndex,
                            0,
                            data.length,
                            packet.sequence / 1000,
                            0);

                    data = null;
                    //decodeDataBuffer.clear();
                    //decodeDataBuffer = null;
                }

                int outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
                        1000);
                do {
                    if (outputBufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER) {
                        //no output available yet
                    } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
                        //encodeOutputBuffers = mDecodeMediaCodec.getOutputBuffers();
                    } else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
                        format = mDecodeMediaCodec.getOutputFormat();
                        //mediaformat changed
                    } else if (outputBufferIndex < 0) {
                        //unexpected result from encoder.dequeueOutputBuffer
                    } else {

                        mDecodeMediaCodec.releaseOutputBuffer(outputBufferIndex,
                                true);

                        outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
                                0);

                    }
                } while (outputBufferIndex > 0);

    }

please dont forget that iFrame (the first frame bytes) contains sensitive data and MUST be fed to the decoder first