FFmpeg：替换实时视频流中的音频

Question

I'm using FFmpeg to encode and live-stream video captured through a DeckLink capture card. The video from the card comes with an audio stream, but I want to replace the audio stream with another. This other audio stream originates from the same source but is ran through an audio processor that adds a fixed delay. The audio is fed back in the pc that runs FFmpeg through a virtual soundcard (audio over IP, but to Windows it looks like a sound card).

I know how to compensate for this fixed delay, but the issue is that audio and video drift slowly out of sync as the stream runs. I'm assuming this is due to the small difference in clock speeds between the virtual soundcard and the DeckLink card.

I've tried the vsync option and the aresample filter in FFmpeg in an attempt to get audio and video to stay synced. However I haven't succeeded in this yet. Is there a way to make FFmpeg resample the audio and/or drop/dup frames in order to get both streams to stay in sync?

Currently I'm running this command, which fails to stay in sync.

ffmpeg.exe -f dshow -i audio="WNIP Input  1 (Wheatstone Network Audio (WDM))" -itsoffset 2.3 -f decklink -thread_queue_size 128 -i "DeckLink SDI (3)" -filter_complex "[1:v:0]bwdif,format=yuv420p,setdar=16/9,scale=-1:576:flags=bicubic[vidout];[0:a:0]aresample=min_comp=0.02:comp_duration=15:max_soft_comp=0.005[audioout]" -c:v libx264 -preset slow -crf 25 -maxrate 1200k -bufsize 2400k -map "[vidout]:0" -map "[audioout]:0" -vsync 1 -r 50 -g 90 -keyint_min 90 -sc_threshold 0 -c:a libfdk_aac -b:a 192k -ac 2 -f flv "rtmp://somewhere"

Answer 1

I found the solution is to use the -use_wallclock_as_timestamps 1 flag for both the audio and the video stream ( ffmpeg docs here ). The complete command in my case then becomes:

ffmpeg.exe -f dshow -use_wallclock_as_timestamps 1 -i audio="WNIP Input  1 (Wheatstone Network Audio (WDM))" -itsoffset 2.1 -f decklink -thread_queue_size 128  -use_wallclock_as_timestamps 1 -i "DeckLink SDI (3)" -filter_complex "[1:v:0]bwdif,format=yuv420p,setdar=16/9,scale=-1:576:flags=bicubic[vidout];[0:a:0]aresample=min_comp=0.02:comp_duration=15:max_soft_comp=0.005[audioout]" -c:v libx264 -preset slow -crf 25 -maxrate 1200k -bufsize 2400k -map "[vidout]:0" -map "[audioout]:0" -vsync 1 -r 50 -g 90 -keyint_min 90 -sc_threshold 0 -c:a libfdk_aac -b:a 192k -ac 2 -f flv "rtmp://somewhere"

Then after a few days both streams are still in sync.