Decode h264 with MediaCodec

Question

I'm decoding a raw h264 received from a wifi cam on android.

Getting the raw packets using a socket and parsing the received data to NAL units.

I have the SPS and PPS units as well (Set as csd-0 and csd-1 in the MediaFormat ).

From all the posts and info I read online I still couldn't get if I'm feeding the decoder the correct data.

This is an example of the resulting video when decoding which apart from the bottom part looks fine.

I've also noticed something odd that when I move the camera the feed seems to run almost completely smooth (no junk at the bottom) and once I set it down the junky video returns (I would have thought it would be the other way around...)

I'm parsing the h264 data to chunks that start with AUD, each chunk starts with AUD and ends when another one begins.

Example:

[0, 0, 0, 1, 9, 48, 0, 0, 0, 1, 6, 1, 9, 0, 12, 8, 36, 104, 0, 0, 3, 0, 1, -128, 0, 0, 0, 1, 33, -32, 96, 97, 92, -97, 71, 89, -31, 127, -120, 11, 23, ..., 0, 0, 1, -64, ... , -59, 2, -32, 62, -111, -64, 0, 0, 1, -32, 0, 0, -124, -128, 5, 33, 0, 1, -60, -31]

[0, 0, 0, 1, 9, 48, 0, 0, 0, 1, 6, 1, 9, 0, 14, 8, 36, 104, 0, 0, 3, 0, 1, -128, 0, 0, 0, 1, 33, -32, 112, 113, 92, -97, 72, 24, 96, 80, 2, 88, 70, ..., 98, -75, 27, 0, 0, 1, -64, 1, 82, ... , 119, 2, -32, 62, -111, -64, 0, 0, 1, -32, 0, 0, -124, -128, 5, 33, 0, 1, -31, 1]

And every few frames I get a chunk with the SPS and PPS

0, 0, 0, 1, 9, 16, 0, 0, 0, 1, 39, SPS, 0, 0, 0, 1, 40, PPS, 0, 0, 0, 1, 6, 0, 13, -128, -89, 89, -128, 8, 117, 0, -89, 89, -128, 8, 117, 64, 1, 9, 0, 16, 8, 36, 104, 0, 0, 3, 0, 1, 6, 1, -60, -128, 0, 0, 0, 1, 37, -72, 1, 0, 1, -1, -16, 13, ... , 108, 60, -83, 101, 0, 0, 1, -32, 0, 0, -124, -128, 5, 33, 0, 3, 25, 65

It was my understanding that each parsed "chunk" (starting with AUD) is an Access Unit and this is what I put in the buffer and feed to the decoder.

Am I feeding the correct input to the decoder?

How could I get rid of the junky video on the bottom?

What could cause the junky part?

I also tried trimming parts of each chunk like trimming the start - [0, 0, 0, 1, 9, 48, 0, 0, 0, 1, 6, 1, 9, 0, 12, 8, 36, 104, 0, 0, 3, 0, 1, -128] and passing only from the 0 ,0 ,0 ,1, 33 part but that didn't make much of a difference.

Answer 1

The "junky part" looks like the encoder padding part. Internally all h264 videos are coded with dimensions being a multiple of 16, but separately flag the exact dimensions of this that should be showed.

You're not describing how you display the decoded data on the screen, if you're using a SurfaceTexture for the output, or manually interpret an YUV buffer.

If manually interpreting buffers, you need to apply the crop fields, as described here: https://developer.android.com/reference/android/media/MediaCodec#data-types

The essential part is this:

The size of the video frame (before rotation) can be calculated as such:

 MediaFormat format = decoder.getOutputFormat(…); int width = format.getInteger(MediaFormat.KEY_WIDTH); if (format.containsKey("crop-left") && format.containsKey("crop-right")) { width = format.getInteger("crop-right") + 1 - format.getInteger("crop-left"); } int height = format.getInteger(MediaFormat.KEY_HEIGHT); if (format.containsKey("crop-top") && format.containsKey("crop-bottom")) { height = format.getInteger("crop-bottom") + 1 - format.getInteger("crop-top"); } 

Answer 2

If you are receiving the stream over RTP/UDP connection, my experience has shown that some packets will be lost over Wifi and may cause some distortions. If you try using RTP over TCP (interleaved format) then you should have a very reliable stream with no packet loss. This could show if the problem is related to your parsing or a problem with packet loss.