无法在华为设备上为 H264 编码视频设置远程 SDP 报价

Question

I'm developing a WebRTC video call app (one-way video and two-way audio) for Android with prebuilt package org.webrtc:google-webrtc:1.0.32006 . I'm stuck with an error from this lib that states:

LS_ERROR
tag: webrtc_video_engine.cc 
message: (line 745): No video codecs supported.

after trying to set a remote SDP offer:

v=0
o=- 955391267744368972 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 audio1 application2
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:NwMv07hmtKD9s8EFikQCvqv7JvMpVhv4
a=ice-pwd:[xxx]
a=rtcp-mux
a=rtcp-rsize
a=sendrecv
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=42c01f;sprop-parameter-sets=J0LAH5WgKA9oQAAAAwBAAAAPOSAACYlgABqz+97gHiRNQA==,KM4fIA==
a=ssrc:2596331602 msid:user823769301@host-4c0ea8a4 webrtctransceiver0
a=ssrc:2596331602 cname:user823769301@host-4c0ea8a4
a=mid:video0
a=fingerprint:sha-256 [xxx]
m=audio 0 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:NwMv07hmtKD9s8EFikQCvqv7JvMpVhv4
a=ice-pwd:[xxx]
a=bundle-only
a=rtcp-mux
a=rtcp-rsize
a=sendrecv
a=rtpmap:97 OPUS/48000/2
a=rtcp-fb:97 nack pli
a=fmtp:97 sprop-maxcapturerate=48000;sprop-stereo=0
a=ssrc:235675792 msid:user823769301@host-4c0ea8a4 webrtctransceiver1
a=ssrc:235675792 cname:user823769301@host-4c0ea8a4
a=mid:audio1
a=fingerprint:sha-256 [xxx]
m=application 0 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:NwMv07hmtKD9s8EFikQCvqv7JvMpVhv4
a=ice-pwd:[xxx]
a=bundle-only
a=mid:application2
a=sctp-port:5000
a=fingerprint:sha-256 [xxx]

It gets me confused as android.media.MediaCodecList contains OMX.google.h264 and HardwareVideoDecoderFactory::supportedCodecs contains name: H264; params: {level-asymmetry-allowed=1, profile-level-id=42e01f, packetization-mode=1} name: H264; params: {level-asymmetry-allowed=1, profile-level-id=42e01f, packetization-mode=1}

I'm using Honor 7X and Huawei P10 and I'm aware that their chipsets lack the hardware H264 encoding which is fine as I do not stream any video from the Android device.

As a video streaming device I use RaspberryPi build with GStreamer as media processing pipeline. The output of v4l2-ctl --list-formats :

    Type: Video Capture

    [0]: 'YU12' (Planar YUV 4:2:0)
    [1]: 'YUYV' (YUYV 4:2:2)
    [2]: 'RGB3' (24-bit RGB 8-8-8)
    [3]: 'JPEG' (JFIF JPEG, compressed)
    [4]: 'H264' (H.264, compressed)
    [5]: 'MJPG' (Motion-JPEG, compressed)
    [6]: 'YVYU' (YVYU 4:2:2)
    [7]: 'VYUY' (VYUY 4:2:2)
    [8]: 'UYVY' (UYVY 4:2:2)
    [9]: 'NV12' (Y/CbCr 4:2:0)
    [10]: 'BGR3' (24-bit BGR 8-8-8)
    [11]: 'YV12' (Planar YVU 4:2:0)
    [12]: 'NV21' (Y/CrCb 4:2:0)
    [13]: 'RX24' (32-bit XBGR 8-8-8-8)

And this is my pipeline definition:

#define RTP_CAPS_OPUS "application/x-rtp,media=audio,encoding-name=OPUS,payload="
#define RTP_CAPS_H264 "application/x-rtp,media=video,encoding-name=H264,payload="

"v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! v4l2h264enc ! video/x-h264,level=(string)3.1,stream-format=(string)byte-stream ! h264parse ! rtph264pay ! "  
        "" RTP_CAPS_H264 "96 ! sendrecv. "
        "alsasrc ! queue ! webrtcdsp noise-suppression-level=3 ! webrtcechoprobe ! audioamplify amplification=1 ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay ! "
        "queue ! " RTP_CAPS_OPUS "97 ! sendrecv. "

My guess is that the H264 profiles are different (RPi: 42c01f ; Huawei: 42e01f ). Also difference in the format's payload might be in a way ( 96 vs. 100 respectively), however I don't have any clue about the significance of these differences.

My second guess is that I should not set a=sendrecv in both clients video media description, but I can't figure out how to accomplish that via WebRTC Android API. Should I put sendonly for RPi then?

And finally, why should I set the VideoEncoderFactory event though I don't encode any video? Without explicitly passing a DefaultVideoEncoderFactory to PeerConnection I cannot establish any connection at all.

Would you be able to point me in the right direction? Any clues, experiences and documentation pieces are highly appreciated!

Answer 1

After some struggle we managed to fix the problem. The solution is to force the sendonly flag on the video media SDP part. This can be done in two ways (in my case both of these were needed to successfully bypass the lacking H264 encoder on Huawei):

'Offer to receive' attribute

When creating a SDP offer you can put an attribute to MediaConstraints that will enforce proper SDP generation

val mediaConstraints = MediaConstraints()     
mediaConstraints.mandatory.add(MediaConstraints.KeyValuePair("OfferToReceiveVideo", "true"))
peerConnection.createOffer(observer, mediaConstraints)

This way the device will generate a list of all video formats that it could decode and put it in SDP video request with recvonly .

---

Gstreamer pipeline definition

To enforce sendonly on gstreamer side we came up with kinda hacky solution

GstSDPMedia *video_media = (GstSDPMedia *)offer->sdp->medias[0].data;
const GstSDPAttribute *direction_attribute = gst_sdp_media_get_attribute(video_media, 5);
gst_sdp_attribute_set(direction_attribute, "sendonly", NULL);

This results in correct attribute being set on SDP offer/answer from gstreamer and does not force the device to look for video encoders.