How to use MediaStreamSource to play h264 frames coming from a matroska file?

Question

I'm trying to render frames coming from an mkv h264 file in silverlight 3 by using the MediaStreamSource. Parsing the mkv file is fine, but I'm struggling with the expected values for CodecPrivateData in SL, which has to be a string, while the PrivateData info from mkv is a binary element. Also, I'm not sure about in which form the frames should be given to SL (ie, the way they are stored in mkv / mp4, or transcoded as NALU) Would anyone have any info on this?

Answer 1

After similar problems of my own and much head-scratching, I am able to answer this question.

• In ReportOpenMediaCompleted(), when setting up your video stream description, you can ignore the CodecPrivateData attribute string, despite what the documentation says. It's not required. (assuming your stream of NAL units includes SPS and PPS units)

• You should send one NAL unit back to the MediaElement for each GetSampleAsync() request. This includes non-picture NAL units, eg SPS / PPS units.

• When you send your NAL units, ensure there are 3-byte start codes (0x00 0x00 0x01) at the beginning of each one. (This is similar to 'Annex B' format, but not quite the same thing)

• In ReportGetSampleCompleted(), set the value of 'Offset' equal to the beginning of the NAL start code, not the actual data. (in most cases this will be zero, assuming you use a fresh stream per NAL unit)

I have blogged a little about the experience here and hope to blog more.

Answer 2

According to the documentation the Codec private data should be set to 00000001 + sps + 00000001 + pps. However the documentation is wrong the value of CodecPrivateData seems to be completely ignored. Instead you need to pass the SPS and PPS NALS (with an annex b header of course) as the first and second result of GetSampleAsync .

For regular media samples normal 4 byte annex b headers headers work just fine

Answer 3

The CodecPrivateData is the contents of the 'avcC' atom which is a child of the 'stsd' atom in an MP4 file. You have to convert the binary data to a string. It will look something like this: "014D401FFFE10017674D401F925402802DD0800000030080000018478C195001000468EE32C8"

You also have to replace the mkv/mp4 lengths to NALU. I've written a little about this (to get Smooth Streaming to work for H.264 files).

Regards,

See: Smooth Streaming H264