GStreamer：通过 UDP 将字符串发送到另一个管道

Question

I have a Gstreamer pipeline in C, meant to send a file to a receiving pipeline via udp.

My sending pipeline is similar to this one:

filesrc location=X.mp4 ! decodebin ! videoconvert ! x264enc ! rtph264pay ! udpsink host=X port=5000

My receiving pipeline is similar to this:

udpsrc port=5000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink

My problem is that I need to send, with each frame, a simple string to the receiving pipeline. I need to be able to to change the string on the fly (by a callback), and my receiving pipeline needs to be able to parse this string (also by a callback).

I understand I can't use textoverlay because the text becomes a part of the video pixels, and the obvious solution seems to be to use subtitles, but I can't figure out how to create a subtitle stream dynamically .
Just to emphasize: I can't use a subtitle file, because I need to be able to send the subtitles "on the fly".

Any help would be GREATLY appreciated.

Answer 1

Eventually I managed to do it.
In case it helps anyone in the future:

In my sender pipeline , I attach a probe to the rtph264pay src element:

rtph264pay_src_pad = gst_element_get_static_pad(rtph264pay_HD, "src");
    gst_pad_add_probe (rtph264pay_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
                       (GstPadProbeCallback) set_x_to_rtp_header, NULL, NULL);
    gst_object_unref (rtph264pay_src_pad);

This is the callback "set_x_to_rtp_header":

static GstPadProbeReturn set_x_to_rtp_header (GstPad          *pad,
                                              GstPadProbeInfo *info,
                                              gpointer         user_data)
{
    GstBuffer *buffer;
    GstRTPBuffer rtpBuffer = GST_RTP_BUFFER_INIT;
    g_print("%s\n", __func__);

    buffer = GST_PAD_PROBE_INFO_BUFFER (info);
    buffer = gst_buffer_make_writable (buffer);

    /* Making a buffer writable can fail (for example if it
     * cannot be copied and is used more than once)
     */
    if (buffer == NULL)
        return GST_PAD_PROBE_OK;

    if (gst_rtp_buffer_map (buffer,GST_MAP_WRITE, &rtpBuffer)) {

        pthread_mutex_lock(&mutex);
        if (gst_rtp_buffer_set_extension_data(&rtpBuffer, setup_data.left_videocrop, sizeof(setup_data.left_videocrop)) != TRUE) {
            g_printerr("cannot add extension to rtp header");
        }
        pthread_mutex_unlock(&mutex);

        gst_rtp_buffer_unmap (&rtpBuffer);
    }

    return GST_PAD_PROBE_OK;
}

In my receiver pipeline , I attach a probe to the rtph264depay sink element:

 rtph264depay_sink_pad = gst_element_get_static_pad(rtph264depay_HD, "sink");
    gst_pad_add_probe (rtph264depay_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
                       (GstPadProbeCallback) get_x_from_rtp_header, videomixer, NULL);
    gst_object_unref (rtph264depay_sink_pad);

And this is the callback "get_x_from_rtp_header":

static GstPadProbeReturn get_x_from_rtp_header (GstPad          *pad,
                                                GstPadProbeInfo *info,
                                                gpointer         user_data)
{
    GstBuffer *buffer;
    GstRTPBuffer rtpBuffer = GST_RTP_BUFFER_INIT;
    guint16 ret_x;
    gpointer data;
    guint wordlen;
    GstElement* videomixer = (GstElement*)user_data;
    GstPad* videomixer_HD_sink_pad;

    g_print("%s\n", __func__);

    buffer = GST_PAD_PROBE_INFO_BUFFER (info);
    buffer = gst_buffer_make_writable (buffer);

    /* Making a buffer writable can fail (for example if it
     * cannot be copied and is used more than once)
     */
    if (buffer == NULL)
        return GST_PAD_PROBE_OK;

    if (gst_rtp_buffer_map (buffer,GST_MAP_READ, &rtpBuffer)) {
        //get x from the rtp header and into ret_x variable
        if (gst_rtp_buffer_get_extension_data(&rtpBuffer, &ret_x, &data, &wordlen) != TRUE) {
            return GST_PAD_PROBE_OK;
        }

        gst_rtp_buffer_unmap (&rtpBuffer);
    }

    return GST_PAD_PROBE_OK;
}