Use MediaCodec and MediaExtractor to decode and code video

Question

I need to decode a video into a sequence of bitmaps, such that I am able to modify them, and then compress them back to a video file in android.

I plan to manage this by using getFrameAtTime and saving it to an image sequence. Then I can modify images in the sequence and code it back to a movie. But I have two problem with this:

• First, as I read it, the getFrameAtTime is for creating thumbnails and will not guarantee returning the correct frame. This makes the video laggy.
• Secondly, saving the images and reading it back takes a long time.

I read that the proper way of doing the decode is with MediaExtractor, this is fine, but I only have examples to render it directly to a surfaceView . Is there any way for me to convert the outputBuffer to a bitmap?

I would need it to get it working with an api level of 16 and above.

Answer 1

You can find a collection of useful examples on the bigflake site.

In particular, the ExtractMpegFramesTest demonstrates how to decode a .mp4 file to Bitmap , and the DecodeEditEncodeTest decodes and re-encodes an H.264 stream, modifying the frames with a GLES shader.

Many of the examples use features introduced in API 18, such as Surface input to MediaCodec (which avoids a number of color-format issues), and MediaMuxer (which allows you to convert the raw H.264 elementary stream coming out of MediaCodec into a .mp4 file). Some devices will allow you to extract video to YUV data in ByteBuffer , modify it, and re-encode it, but other devices extract to proprietary YUV color formats that may be rejected by the API 16 version of MediaCodec .

I'd recommend coding for API 18 (Android 4.3 "Jellybean" MR2) or later.

Answer 2

There are many people saying that the method onFrameAvailable() is never called. Well, the listener should be in a different thread than the main thread. To set the listener do this: (where this is the class listener that implements SurfaceTexture.IOnFrameAvailableListener ):

mSurfaceTexture.SetOnFrameAvailableListener(this);