video capturing with V4l2 has dropped/lost frames

Question

I have a scientific application which captures a video4Linux video stream. It's crucial that we capture each frame and no one gets lost. Unfortunately frames are missing here and there and I don't know why.

To detect dropped frames I compare the v4l2_buffer's sequence number with my own counter directly after reading a frame:

void detectDroppedFrame(v4l2_buffer* buffer) {
        _frameCounter++;

        auto isLastFrame = buffer->sequence == 0 && _frameCounter > 1;
        if (!isLastFrame && _frameCounter != buffer->sequence+1)
        {
                std::cout << "\n####### WARNING! Missing frame detected!" << std::endl;               
                _frameCounter = buffer->sequence+1; // re-sync our counter with correct frame number from driver.
        }
}

My running 1-file example gist can be found at github (based on official V4L2 capture example): https://gist.github.com/SebastianMartens/7d63f8300a0bcf0c7072a674b3ea4817

Tested with webcam on Ubuntu 18.04 virtual machine on notebook-hardware (uvcvideo driver) as well as with CSI camera on our embedded hardware running ubuntu 18.04 natively. Frames are not processed and buffers seems to be grabbed fast enough (buffer status checked with VIDIOC_QUERYBUF which shows that all buffers are in the driver's incoming queue and V4L2_BUF_FLAG_DONE flag is not set). I have lost frames with MMAP as well as with UserPtr method. Also it seems to be independent of pixelformat, image size and framerate!

To me it looks like if the camera/v4l2 driver is not able to fill available buffers fast enough but also increasing the file descriptor priority with VIDIOC_S_PRIORITY command does not help (still likely to be a thread scheduling problem?).

=> What are possible reasons why V4L2 does not forward frames (does not put them into it's outgoing queue)? => Is my method to detect lost frames correct? Are there other options or tools for that?

Answer 1

I had a similar problem when using the bttv driver. All attempts to capture at full resolution would result in dropped frames (usually around 10% of frames, often in bursts). Capturing at half resolution worked fine.

The solution that I found, though far from ideal, was to apply a load to the linux scheduler. Here is an example, using the "tvtime" program to do the capture:

#! /bin/bash
# apply a load in the background
while true; do /bin/true; done &> /dev/null &
# do the video capture
/usr/bin/tvtime "$@"
# kill the loop running in the background
pkill -P $$

This creates a loop that repeatedly runs /bin/true in the background. Almost any executable will do (I used /bin/date at first). The loop will create a heavy load, but with a multi-core system there is still plenty of headroom for other tasks.

This obviously isn't an ideal solution, but for what it's worth, it allows me to capture full-resolution video with no dropped frames. I have little desire to poke around in the driver/kernel code to find a better solution.

FYI, here are the details of my system:

OS:  Ubuntu 20.04, kernel 5.4.0-42
MB:  Gigabyte AB350M-D3H
CPU: AMD Ryzen 5 2400G
GPU: AMD Raven Ridge

Driver name      : bttv
Card type        : BT878 video (Hauppauge (bt878))
Bus info         : PCI:0000:06:00.0
Driver version   : 5.4.44
Capabilities     : 0x85250015