FFMPEG 将来自 image2pipe 的视频与来自 RTMP 的音频同步

Question

I have an rtmp url, and i need to put overlay using opencv and keeping audio.

It works well, exept for syncronization between audio and video.

with this command

ffmpeg -re -y -f image2pipe -i - -i rtmp://192.168.178.32:1935/opencv-hls/test -map 0:v:0 -vcodec libx264 -g 50 -keyint_min 50 -map 1:a:0 -b:a 128k -f mpegts -codec:v mpeg1video -b:v 700k -s 1024x576 -bf 0 -q 1 http://localhost:3000/mystream

I open an image2pipe where i send my frame with overlay, and i use the same input of opencv as second command for ffmpeg to acquire audio and send all to mpegts url

This is the ffmpeg output

ffmpeg version 4.1.4-1build2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.2.1-4ubuntu1)
  configuration: --prefix=/usr --extra-version=1build2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, image2pipe, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1024x576 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #1, flv, from 'rtmp://192.168.178.32:1935/opencv-hls/test':
  Metadata:
    Server          : NGINX RTMP (github.com/arut/nginx-rtmp-module)
    displayWidth    : 1024
    displayHeight   : 576
    fps             : 0
    profile         : 
    level           : 
  Duration: 00:00:00.00, start: 4.639000, bitrate: N/A
    Stream #1:0: Audio: aac (LC), 44100 Hz, stereo, fltp
    Stream #1:1: Video: h264 (Constrained Baseline), yuv420p(progressive), 1024x576, 29.58 fps, 29.58 tbr, 1k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> mpeg1video (native))
  Stream #1:0 -> #0:1 (aac (native) -> mp2 (native))
[swscaler @ 0x55f256d63980] deprecated pixel format used, make sure you did set range correctly
Output #0, mpegts, to 'http://localhost:3000/mystream':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: mpeg1video, yuv420p(progressive), 1024x576 [SAR 1:1 DAR 16:9], q=2-31, 700 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.35.100 mpeg1video
    Side data:
      cpb: bitrate max/min/avg: 0/0/700000 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: mp2, 44100 Hz, stereo, s16, 128 kb/s
    Metadata:
      encoder         : Lavc58.35.100 mp2

I always have a delay between audio and video and it increese with time. Do you have any idea on how to sync them?

thanks Andrea

Answer 1

[sorry for spelling mistakes, i am just not bothered to fix it]

Not sure if it is still needed, but I had a similar problem; the audio was starting 1 sec behind the video and drifting further over time.

The drifting was fixed after I inserted -vsync 2 so the video will drop frames if a duplicate timestamp occured. (this might also be a framerate issue, use -vsync 1 instead, to drop or duplicate frames to achieve the requested fps, or use -vsync -1 if you want ffmpeg to auto choose between 1 or 2), for me it was not the audio that was causing the seemingly drift. It was the occasion frame loss that caused the frame timestamps to mess up.

Even tough this might solve timestamp issues, there is a possibility that indeed the audio itself is drifting, in this case -af aresample=async=1 -adrift_threshold <time-in-sec> might do the trick, ffmpeg will drop samples in order for the timestamps to match up, lower values will do a hard (drop/add) of frames and higher values will do a soft (squeez/stretch) of frames.

if there is still the 1sec drift at the beginning you can use -itsoffset <value> before the audio input parameter to manually correct it (or dynamically with personnalised code), this will initiate the the stream with an offset, it can also be a negative value to do a reverse offset, making the input start earlier in comparisson with the rest. So in your case it would be -itsoffset -1 .

-frame_drop_threshold <nr. of frame-offset> -frame_drop_threshold <nr. of frame-offset> might also be an interesting parameter to check out, but i would use the audio timestamps as main timestamp in this case, since the videoframes might duplicate their timestamps.

https://ffmpeg.org/ffmpeg-all.html#Advanced-options [ Advanced-Options 5.11 ]