FFmpeg v4l2 复制不稳定

Question

Prelude

I am using ffmpeg 4.2.2 on an Ubuntu 20.04 machine to clone the feed of a USB webcam ( v4l2 device as /dev/video0 ) so that multiple applications may access the feed. I used this ffmpeg command which worked lovely:

ffmpeg -f v4l2 -i /dev/video0 -codec copy -f v4l2 /dev/video1

Essentially this uses a v4l2 loopback device (dummy device) in form of /dev/video1 and a consuming application simply sees this as a regular v4l2 device.

After that worked well, I ran into the need to change the output pixel format (related SO question ). My source device provides yuv420p and the sink application can only deal with yuyv422 . To achieve this, I use the -pix_fmt argument. This is however not supported when using -codec copy so I need to specify the codec explicitly. This is no problem as v4l2 devices only support raw video (at least as per my knowledge). Therefore, I end up with this:

ffmpeg -f v4l2 -i /dev/video0 -c:v rawvideo -pix_fmt yuyv422 -f v4l2 /dev/video1

The problem

Unfortunately, this appears to be very unstable. In about 1 out of 10 times of launching this command it works as expected: ffmpeg starts cloning the feed and converts the color format:

joel@joel-ubuntu:~$ ffmpeg -f v4l2 -i /dev/video0 -c:v rawvideo -pix_fmt yuyv422 -f v4l2 /dev/video1
ffmpeg version 4.2.2-1ubuntu1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-3ubuntu1)
  configuration: --prefix=/usr --extra-version=1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[video4linux2,v4l2 @ 0x55a4f38a5700] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 11224.864514, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, 14.33 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, video4linux2,v4l2, to '/dev/video1':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 640x480, q=2-31, 70451 kb/s, 14.33 fps, 14.33 tbn, 14.33 tbc
    Metadata:
      encoder         : Lavc58.54.100 rawvideo
frame=  214 fps= 15 q=-0.0 Lsize=N/A time=00:00:14.93 bitrate=N/A dup=1 drop=0 speed=1.05x    
video:128400kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.

However, the other times I get very archaic behavior where the output is never usable after except for the first frame (then it freezes) and ffmpeg goes haywire mentioning over 100k duplicate frames in just a few second as well as reporting FPS way beyond 30'000. After a few seconds it then crashes or it simply hangs until I Ctrl^C it:

joel@joel-ubuntu:~$ ffmpeg -f v4l2 -i /dev/video0 -c:v rawvideo -pix_fmt yuyv422 -f v4l2 /dev/video1
ffmpeg version 4.2.2-1ubuntu1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-3ubuntu1)
  configuration: --prefix=/usr --extra-version=1ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[video4linux2,v4l2 @ 0x561ef693c700] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 11156.178415, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, 1000k tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[video4linux2,v4l2 @ 0x561ef6940400] Frame rate very high for a muxer not efficiently supporting it.
Please consider specifying a lower framerate, a different muxer or -vsync 2
Output #0, video4linux2,v4l2, to '/dev/video1':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 640x480, q=2-31, 4915200000 kb/s, 1000k fps, 1000k tbn, 1000k tbc
    Metadata:
      encoder         : Lavc58.54.100 rawvideo
More than 1000 frames duplicated
More than 10000 frames duplicated
More than 100000 frames duplicated=N/A time=00:00:00.15 bitrate=N/A dup=151050 drop=0 speed=0.0456x    
27653321 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=0 speed=0.0289x    
27733577 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=1 speed=0.0256x    
27787315 frame duplication too large, skipping
27859335 frame duplication too large, skipping
27941595 frame duplication too large, skipping
28006927 frame duplication too large, skipping
28069615 frame duplication too large, skipping
28140781 frame duplication too large, skipping
28217871 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=8 speed=0.0251x    
28282039 frame duplication too large, skipping
28347777 frame duplication too large, skipping
28449099 frame duplication too large, skipping
28491015 frame duplication too large, skipping
28565585 frame duplication too large, skipping
28630457 frame duplication too large, skipping
28710537 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=15 speed=0.0247x    
28770549 frame duplication too large, skipping
28843129 frame duplication too large, skipping
28922285 frame duplication too large, skipping
28985573 frame duplication too large, skipping
29061631 frame duplication too large, skipping
29133801 frame duplication too large, skipping
29193197 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=22 speed=0.0243x    
29269443 frame duplication too large, skipping
29349681 frame duplication too large, skipping
29403629 frame duplication too large, skipping
29489299 frame duplication too large, skipping
29542137 frame duplication too large, skipping
29611859 frame duplication too large, skipping
29688271 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=29 speed=0.0238x    
29759777 frame duplication too large, skipping
29824489 frame duplication too large, skipping
29897671 frame duplication too large, skipping
29966443 frame duplication too large, skipping
30037557 frame duplication too large, skipping
30117317 frame duplication too large, skipping
30172847 frame duplication too large, skipping
30255657 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=37 speed=0.0234x    
30331325 frame duplication too large, skipping
30385245 frame duplication too large, skipping
30458209 frame duplication too large, skipping
30532393 frame duplication too large, skipping
30596453 frame duplication too large, skipping
30681671 frame duplication too large, skipping
30740757 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=44 speed=0.023x    
30806259 frame duplication too large, skipping
30873657 frame duplication too large, skipping
30950843 frame duplication too large, skipping
31013967 frame duplication too large, skipping
31099321 frame duplication too large, skipping
31172533 frame duplication too large, skipping
31224713 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=51 speed=0.0227x    
31297053 frame duplication too large, skipping
31378711 frame duplication too large, skipping
31438565 frame duplication too large, skipping
31506345 frame duplication too large, skipping
31589213 frame duplication too large, skipping
31656175 frame duplication too large, skipping
31718295 frame duplication too large, skipping
31814019 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=59 speed=0.0223x    
31876469 frame duplication too large, skipping
31917425 frame duplication too large, skipping
31995521 frame duplication too large, skipping
32062573 frame duplication too large, skipping
32127183 frame duplication too large, skipping
32197109 frame duplication too large, skipping
32267481 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=66 speed=0.022x    
32358605 frame duplication too large, skipping
32407853 frame duplication too large, skipping
32478077 frame duplication too large, skipping
32523429 frame duplication too large, skipping
32557449 frame duplication too large, skipping
32587093 frame duplication too large, skipping
32620687 frame duplication too large, skipping
32654747 frame duplication too large, skipping
32689485 frame duplication too large, skipping
32719435 frame duplication too large, skipping
32753423 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=77 speed=0.0216x    
32781827 frame duplication too large, skipping
32820747 frame duplication too large, skipping
32860855 frame duplication too large, skipping
32884671 frame duplication too large, skipping
32918767 frame duplication too large, skipping
32955299 frame duplication too large, skipping
32989505 frame duplication too large, skipping
33019757 frame duplication too large, skipping
33053493 frame duplication too large, skipping
33082955 frame duplication too large, skipping
33116965 frame duplication too large, skipping
33156613 frame duplication too large, skipping
33188409 frame duplication too large, skipping
33224915 frame duplication too large, skipping
33255715 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=92 speed=0.0213x    
33292845 frame duplication too large, skipping
33323677 frame duplication too large, skipping
33352059 frame duplication too large, skipping
33386205 frame duplication too large, skipping
33420195 frame duplication too large, skipping
33455171 frame duplication too large, skipping
33484311 frame duplication too large, skipping
33520791 frame duplication too large, skipping
33550153 frame duplication too large, skipping
33587191 frame duplication too large, skipping
33623055 frame duplication too large, skipping
33656991 frame duplication too large, skipping
33686655 frame duplication too large, skipping
33720711 frame duplication too large, skipping
33749987 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=107 speed=0.021x    
33784387 frame duplication too large, skipping
33823663 frame duplication too large, skipping
33869275 frame duplication too large, skipping
33907671 frame duplication too large, skipping
34043059 frame duplication too large, skipping
34089679 frame duplication too large, skipping
34120031 frame duplication too large, skipping
34193323 frame duplication too large, skipping
34263479 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=116 speed=0.0207x    
34403751 frame duplication too large, skipping
34470319 frame duplication too large, skipping
34542507 frame duplication too large, skipping
34613587 frame duplication too large, skipping
34687495 frame duplication too large, skipping
34755115 frame duplication too large, skipping:00:00.70 bitrate=N/A dup=707630 drop=122 speed=0.0204x    
34823847 frame duplication too large, skipping
34893931 frame duplication too large, skipping
34963903 frame duplication too large, skipping
35034011 frame duplication too large, skipping
35103715 frame duplication too large, skipping
35178691 frame duplication too large, skipping
frame=707641 fps=20090 q=-0.0 Lsize=N/A time=00:00:00.70 bitrate=N/A dup=707630 drop=129 speed=0.0201x    
video:424584600kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.

I have no idea what leads to this behavior nor how to track it down & fix it. Does anybody have experience with this type of behavior or is there a simple issue on my setup?

Note: I am using DroidCam. DroidCam provides a Linux client that in turn provides a v4l2 output device. Other than a few minor hicups I did not notice any behavior that would be different from a regular webcam. Unfortunately I can't reproduce this with an ordinary USB UVC webcam as the webcam already provides the desired output pixel format. This output pixel format conversion is really just necessary because DroidCam outputs yuv420p but the application consuming the stream can only handle yuyv422 .

Answer 1

The V4L2 device is a constant frame rate writer, so ffmpeg will duplicate frames to keep a constant frame rate if source frame rate is lower. In the absence of a user-set rate, ffmpeg sets it to the input's frame rate. For capture devices which don't signal a framerate explicitly, ffmpeg infers the rate by looking for the smallest interval between frames.

When ffmpeg detects 1000k tbr , that's when you get mass duplication. Either set a sane number -r 15 or disable frame duplication, -vsync vfr .