How do I render “Video” in Qt / X11 on an embedded device

Question

I am currently trying to write an application that displays live "video" from a specialized USB device. The USB device provides approx. 10 frames/sec of 320x240 video in an array of gray levels. The application is written in Qt on an iMx51 (800Mhz arm) platform running linux and X11. My application needs to display this video, scaled by a factor of 2 (so 640x480) and use as little CPU as possible.

Currently, the images are displayed as a simple QLabel which updates it's pixmap every time a new video frame has been transmitted from the device. When the QLabel is being displayed at 320x240 the video plays smoothly and the application uses 20% of the CPU.

When we attempt to scale the QLabel to 640x480 in size the CPU usage spikes, consuming all available resources at 10frames/sec. Our only means of reducing the CPU usage is to drop frames.

We have tried writing a widget with a custom paint event to paint the contents of the video image ourselves to no avail, this technique used slightly more CPU than the QLabel.

Our other products that do this have the capability to blit the image directly to a raw frame buffer (no x11) and despite their much slower CPU's they can handle this task without any problem. It seems that X11 and Qt is adding an immense amount of overhead for us.

There must be a PROPER way to display a constantly updating image through Qt and X11 that is not so resource intensive. We are exploring the possibility of getting openGL ES up and running on the unit but that may be out of the question for the time being. Is there something I'm missing in the Qt Framework that will allow me to accomplish this relatively simple task?

Answer 1

After downloading and reading through the VLC source code I settled on using the Xvideo extension for X11. The VLC source code contains an xvideo output "engine" using xcb to communicated with the xserver. I used this as an example to code my own solution.

On our platform the scaling and display features of the xvideo extension are hardware accelerated.

I intend to provide a tutorial and example code as soon as time allows.

Answer 2

Your problem with resolution is that the CPU is far much slower than the GPU, therefore rendering the frames with labels is really a bad idea. When you increase the resolution, you exponentially increase the size of the data that the CPU will have to work on.. When working with video you have to use the GPU which runs much faster.

I suggest you to use the QGL widget from the Qt OpenGL module ( http://doc.qt.nokia.com/4.7/qtopengl.html ).

You can read the frames from your device, move them to the video memory, use the GPU to render them (you can also apply filters and effects if you learn/know how to do it), saving a lot of CPU time and decreasing your resolution problems.

Answer 3

Do not use a QLabel to display moving contents. Instead, sub-class QWidget and implement your own widget. Also, you might want to experiement with direct painting (flags for QWidget) to avoid double buffering, etc.

Of course, using a GPU for scaling and such is good, but it sounds to me as if you still have to move all the data from the device to the GPU using the CPU. Ie you will still be using quite a few cycles just to move ones and zeros.

Answer 4

"Plain" X11 is very inefficient for bitmap rendering due to its client-server architecture, you need to use MIT-SHM extension. Depending on the Qt version, it may include "raster" graphic engine which is using MIT-SHM natively.

See also What is the fastest way to display an image in QT on X11 without OpenGL? - eventually I managed to get very decent performance from Qt. The trick is to watch how many times you copy your bitmap and try to minimize it.

Note that you don't really need OpenGL for bitmaps, you could try 2D acceleration if you have a suitable driver for your hardware. You have shared memory, so for bitmap rendering your GPU is as fast as CPU, it would just save you some CPU cycles if you do bitmap rendering (essentially, a memory to memory transfer) using some other hardware.