Fastest way to Decode and Display many H264 Videos Simultaneously C#

Question

As you might surmise from the question title, we are required to decode and display multiple (eg, eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.

I'm going to provide a bit of background before I get to the crux of the problem.

The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.

We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.

An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).

The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.

The millisecond we started drawing to an HWND created on the UI thread (eg, a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.

Gah! CanOfWorms.Open().

Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows. Solution: Place the video windows Always On Top.

Problem: When the user switches to another application, the video windows are still on top. Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.

Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!" Solution: Tell user to shut up and that Word sucks anyways.

Problem: I get fired.

etc. etc.

I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.

Any thoughts/suggestions? Am I completely out to lunch here?

I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).

Answer 1

Forget about BitBlt-ing and do this:

• for each window you want your video to be played, create one DirectShow graph and attach the renderer of the graph to that window
• before renderer in the graph put the samplegrabber filter. It will allow you to have callback in which you'll be able to fill the buffer
• instead of blitting, decode to the buffer provided in samplegrabber.

In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.

Use DirectShowNet library.

EDIT:

And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.

YET ANOTHER EDIT:

BitBlts are ALWAYS serialized, ie they can't run in parallel.

Answer 2

The millisecond we started drawing to an HWND created on the UI thread (eg, a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.

What kind of funky behavior? If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation? Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)

I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.

Don't do that. Try to draw the received data in your c# application. In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.

Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).

I hope this helps, but also need more information about the problem...

Answer 3

I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:

While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos

My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.