How to use FFMPEG with Nvidia Acceleration (Nvenc) to hardcode / burn in subtitles

Question

So I'm trying to use Nvenc to accelerate video encoding. The aim is to have 1 input video file and 1 input subtitle, and to get multiple outputs at different resolutions with subtitles hardcoded or burned into the video. I've tried multiple approaches but can't figure out how to do it.

Here's the command that I'm currently using:

ffmpeg -hwaccel cuvid -i 3030025890-TEST.mp4 -i output_ar.srt  -filter_complex "[0:v]scale_npp=1920:1080, hwdownload,format=nv12[base], [base]subtitles=output_ar.srt[marked]" -map "[marked]" -c:v h264_nvenc -map 0:v:0 -map 0:a:0 -g 50 -b:v 5M -maxrate 5.5M -minrate 4M -bufsize 5M -preset fast 1080_output.mp4

and here's the output:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '3030025890-TEST.mp4':
  Metadata:
    major_brand     : M4V 
    minor_version   : 1
    compatible_brands: isomavc1mp42
    creation_time   : 2021-01-05T13:45:58.000000Z
  Duration: 00:45:04.28, start: 0.000000, bitrate: 5574 kb/s
    Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 5000 kb/s, 25 fps, 25 tbr, 25k tbn, 50 tbc (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Video Media Handler
      encoder         : Elemental H.264
    Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
    Stream #0:2(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
    Stream #0:3(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
    Stream #0:4(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
    Stream #0:5(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
    Stream #0:6(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default)
    Metadata:
      creation_time   : 2021-01-05T13:45:58.000000Z
      handler_name    : ETI ISO Audio Media Handler
Input #1, srt, from 'output_ar.srt':
  Duration: N/A, bitrate: N/A
    Stream #1:0: Subtitle: subrip
[Parsed_subtitles_3 @ 0x5601070b1dc0] Shaper: FriBidi 0.19.7 (SIMPLE)
Fontconfig error: Cannot load default config file
[Parsed_subtitles_3 @ 0x5601070b1dc0] No usable fontconfig configuration file found, using fallback.
Fontconfig error: Cannot load default config file
[Parsed_subtitles_3 @ 0x5601070b1dc0] Using font provider fontconfig
Stream mapping:
  Stream #0:0 (h264) -> scale_npp (graph 0)
  subtitles (graph 0) -> Stream #0:0 (h264_nvenc)
  Stream #0:0 -> #0:1 (h264 (native) -> h264 (h264_nvenc))
  Stream #0:1 -> #0:2 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x56010792f980] Error creating a NVDEC decoder: 1
[h264 @ 0x56010792f980] Failed setup for format cuda: hwaccel initialisation returned error.
[Parsed_subtitles_3 @ 0x560107364cc0] Shaper: FriBidi 0.19.7 (SIMPLE)
Fontconfig error: Cannot load default config file
[Parsed_subtitles_3 @ 0x560107364cc0] No usable fontconfig configuration file found, using fallback.
Fontconfig error: Cannot load default config file
[Parsed_subtitles_3 @ 0x560107364cc0] Using font provider fontconfig
Impossible to convert between the formats supported by the filter 'graph 0 input from stream 0:0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
[aac @ 0x56010734d400] Qavg: 65536.000
[aac @ 0x56010734d400] 2 frames left in the queue on closing
Conversion failed!

Edit: Made some progress. I now no longer get an error, but upon viewing the output, there's no subtitles burned in. Command:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -i 3030025890-TEST.mp4  -c:v h264_nvenc -map 0:v:0 -map 0:a:0 -g 50 -b:v 5M -maxrate 5.5M -minrate 4M -bufsize 5M -vf "scale_npp=1920:1080, hwdownload, format=nv12, subtitles=output_ar.srt, hwupload" -preset fast 1080_output.mp4

Answer 1

Ok, so I've managed to resolve the issue. I was running ffmpeg inside a docker container and that presented its own set of issues.

1. If you're using linux and see something such as "Fontconfig error: Cannot load default config file" then make sure you have fontconfig installed. On Ubuntu:

 sudo apt-get update -y sudo apt-get install -y fontconfig-config

If like me you're using a docker container and got the above error, then make sure to add the following commands to your dockerfile:

RUN apt-get update -y
RUN apt-get install -y fontconfig-config

2. If you're planning on running any filters on your CPU (some filters such as subtitles and libass are unsupported on the GPU / using hardware acceleration), then you'll have to download the frames (only if using the -hwaccel flag) after decoding onto your RAM before applying the filters and re-uploading to the GPU VRAM. In my case I won't be using the -hwaccel flag as this makes it simpler to run CPU bound filters as per Nvidia's documentation . In my tests, it also makes things a bit faster. The filter command then becomes:

-vf "subtitles=mysubtitles.srt, hwupload_cuda, scale_npp=1920:1080"

3. Combine the above points and you arrive at the following command. Note the absence of the -hwaccel flag.

ffmpeg -threads 16 -vsync 0 -c:v h264_cuvid -i 3030025890-TEST.mp4 -c:v h264_nvenc -vf "subtitles=mysubtitles.srt, hwupload_cuda, scale_npp=1920:1080" 1080p_output.mp4 -vf "subtitles=mysubtitles.srt, hwupload_cuda, scale_npp=1280:720" -c:v h264_nvenc -preset fast 720p_output.mp4

Edit: Wasn't satisfied with the performance of the above command, so I spent some time trying to optimize it. What I've discovered is that you can use the split filter to split out your output stream. What I've essentially done is decode using the GPU, then download to CPU memory, apply the filters using the CPU once, upload it to the GPU, split out the output stream to 2 (one for the 1080p output and another for the 720p) and then let the GPU handle scaling + encoding. This yielded a 5x increase in encoding speed when compared to the previous command.

This is the command that I'm currently using (note that there are other flags such as -g for keyframe interval and -b:v for setting the target bitrate, which depending on your use case might not be necessary):

ffmpeg -i Input.mp4 -c:v h264_nvenc -map 0:a:0 -g 50 -b:v 5.0M -maxrate 5.5M -minrate 4.2M -bufsize 4.5M -filter_complex "[0:v]subtitles="mySubs.srt",hwupload_cuda,split=2[c0][c1];[c0]scale_npp=1920:1080[1080];[c1]scale_npp=1280:720[720];" -map "[1080]" -preset fast "1080p_output.mp4" -c:v h264_nvenc -map 0:a:0 -g 50 -b:v 3.4M -maxrate 3.7M -minrate 2.8M -bufsize 3.1M -map "[720]" -preset fast "720p_output.mp4"