How to use FFmpeg H264 encoder in WebRTC?

Question

For H264 encoding WebRTC uses OpenH264 which does not support hardware acceleration. There are many third party codecs included in WebRTC including WebRTC. How FFmpeg can be used instead? "is_component_ffmpeg=true" does not seem to do anything.

The goal here is to encode with hardware acceleration to have reduced latency and cpu usage. We have hardware encoder running but do not know how to plug that into webrtc. Using hardware acceleration is the closest option.

Where do we need to look at to use FFmpeg? or use externally encoded h264 data stream?

Answer 1

We ended up modifying h264_encoder_impl by replacing all OpenH264 API calls with our own encoder calls.

WebRTC constantly keeps asking the encoder implementation to update the bitrate and framerate as it sees fit for current available bandwidth. The HW encoder we used supported updating only bitrates on the fly and that worked fine with WebRTC. Framerate was set to a fixed value.

As we did not change framerate as per wishes of WebRTC and it still worked fine, I think that encoded stream can also be sent the same way after doing only RTPFragmentation properly for given encoded buffer.

Answer 2

We've attempted to shunt the encoding portion of the WebRTC project in the past with little luck (we wanted to pass through data that had already been encoded to multiple WebRTC clients). My impression is that it's very tightly integrated with quality of service. WebRTC wants to adjust the encoder settings based on current network traffic.

The best solution we found is to actually roll your own WebRTC using the dtlssrtpenc , nicesink , and nicesrc elements from the OpenWebRTC project:

https://github.com/EricssonResearch/openwebrtc-gst-plugins

This wasn't at all easy to do. WebRTC has a very complicated handshake and those GStreamer elements require a lot of special hookup, but it did yield the desired results.

Oh and btw our experience is that openh264 works quite well for WebRTC traffic and we ended up using it for many cases.