gstreamer webrtc h264 playback stops after few frames in browser

Question

I need help for debugging a probabilistic issue. I built a gstreamer pipeline to stream NVENC encoded h264 bitstreams(video only) to browser. Browser seldom plays properly. In most cases only few frames are rendered then the picture gets frozen.

The NVENC settings follow "https://cloud.google.com/solutions/gpu-accelerated-streaming-using-webrtc" which are h264 high profile & low latency high quality & NVENC_INFINITE_GOPLENGTH(some settings have been tried, like rateControlMode/enableVFR/sliceMode/repeatSPSPPS/outputAUD but no help). At runtime, NVENC encodes real-time rendered opengl fbo texture to h264 bitstreams and push them into gstreamer via appsrc. Currently the texture size is 512x512 and fed at 10/20/30 fps.

I use gstreamer 1.18.2, the pipeline is defined as "appsrc name=nvenc_src do-timestamp=1, video/x-h264, stream-format=byte-stream, alignment=au, rtph264pay aggregate-mode=zero-latency, queue. application/x-rtp,media=video,encoding-name=H264,payload=123 ! webrtcbin bundle-policy=max-compat name=backend_webrtc".

The gstreamer part codes follow the sendrecv example(replacing libsoup with websocketpp and removing the recv logics).

The application is built as MSVC 2019 32-bit. The browser decoder is NVDEC. Exe application and js codes run on the same PC(windwos 10, gtx1060, driver version 460.89). I've tried in Chrome(87.0.4280.88) and edge(87.0.664.66). I also tried running the js codes in android(chrome) and ios(safari) and get the same results.

It can be concluded that NVENC generates 'correct' h264 bitstreams. I dump the raw h264 bitstreams into file. The file plays properly in VLC. I also tried pushing the dumped h264 bitstreams into gstreamer. The frozen issue still happens.

After the picture is frozen, the playback never recovers. the browser's 'webrtc-internals' shows that bytes/headerBytes/packests_Received keep growing, while frameReceived/framesDecoded/framesDropped stay unchanged.

Since the bitwise same h264 frames behave differently at different runs, I guess rtp timestamps might cause the issue. I've tried setting appsrc's do-timestamp to 0 and manually set gstbuffer's PTS but it does not help.

Answer 1

Here are few things that you need to pay attention to:

• Infinite GOP will not work - you must configure NVENC to send a key frame every 30 - 60 frames.
• Of course SPS-PPS NALs must come before each key frame.
• Prohibit B-frames: WebRTC doesn't support them because they increase latency.
• Startup codes between NALs must be 3-bytes startup codes: WebRTC doesn't respect 2-bytes startup codes. We bumped into this issue before and had to manually correct the startup codes.

Answer 2

Thanks to user1390208's kind reminds, I use an h264 analyzer tool to check the dumpped bitstreams and find the evils.

Browser does support infinite GOP. But it needs keyframe & SPS-PPS for recovering from error. The need for resend arises with a high probability during the launch procedure. So a quick solution is resending keyframe & SPS-PPS after js detects fps is 0 and send a resend request to gstreamer via the webrtc data channel.

The reasons I failed to find the answer are two:

• I didn't set encodePicFlags before calling nvEncEncodePicture. NVENC always generates infinite GOP whether gopLength & frameIntervalP are set to all I or I&P. There are many GOP related parameters and look confusing to me now. In my current codes, the only way to get the desried GOP control is by setting NV_ENC_PIC_PARAMS::encodePicFlags before calling nvEncEncodePicture. Note that I use NV_ENC_PRESET_LOW_LATENCY_HQ_GUID & NV_ENC_H264_PROFILE_HIGH_GUID which might cause infinite GOP is always generated(when encodePicFlags is not set)? NVENC reports no error when settings gopLength & frameIntervalP & repeatSPSPPS, so I thought the issue also happens when the GOP is all of I frames and SPS-PPS does not help.

• Infinity GOP does not always cause the mentioned issue in browser during the launch procedure.

So before I check the h264 bitstreams with analyzer tool, it looks to me that even all-keyframe+SPS-PPS bitstreams have this probability issue.

BTW, NVENC generates 4-byte start code.