Gstreamer 定制管道不播放/掛出

Question

上下文

我正在嘗試構建一個小型視頻剪輯器來處理具有多個音軌的視頻文件，例如，一個用於語音聊天，一個用於麥克風，一個用於游戲中的音頻。 最終目標是能夠導出所有音軌混合在一起的剪輯，並最終能夠改變每個音軌的音量。

我正在使用 Qt 構建 UI。

在構建渲染/導出部分之前，我嘗試使用 GStreamer 在我的應用程序中構建一個小型多軌音頻視頻查看器。

流水線

通過查看 GStreamer 教程，我構建了一個繼承自 QObject 的 PlayerPipeline class（帖子中的代碼如下）。 這個 class 以文件路徑和 QWidget 作為參數，並從 UriDecodeBin 構建管道，以達到此形狀（此處視頻中有 2 個音軌）：

|              | -> video  ->  | Queue | ---------------------------------------> | PlaySink |
| UriDecodeBin | -> audio1 ->  | Queue | -> | AudioConvert | -> | AudioMixer | -> |          |
|              | -> audio2 ->  | Queue | -> | AudioConvert | -> |            |

（我不確定這里是否需要隊列，因為如果我沒記錯的話，UriDecodeBin 已經在為其每個 src 墊創建隊列）

通過以下步驟：

• 創建管道和總線
• 創建UriDecodebin，加入pipeline並播放pipeline
• 在 UriDecodeBin 上設置 no-more-pad 回調，一旦觸發，查詢每個 src pad 並用探針阻止它們
• 一旦所有焊盤都被阻塞，暫停管道並構建管道的 rest
• 然后執行刷新搜索（據說觸發探針回調並刪除探針，但不知何故不會發生）以設置管道准備播放

問題

有兩件事發生了。

如果我在預gst_element_get_state后嘗試 gst_element_get_state，沒有超時限制，它有時會無限期掛起。

如果我不調用它，但嘗試在播放 state 時設置管道，則什么也不會發生，而且我也不會在總線上收到 state 更改消息。

我試圖添加隊列（根據另一篇 SO 帖子），但我承認我現在有點不知所措。 任何幫助，甚至是嘗試解決問題/獲得洞察力的想法，都將不勝感激。

代碼

請注意這里沒有 memory 管理，所以代碼很可能充滿漏洞，這很正常，我想在添加清理/清理錯誤管理代碼之前擁有播放器的工作版本。

這個文件gstmanips.h定義了幾個助手，實現起來可能很簡單，所以我在這里不詳細介紹它們。

bool isPadSrc(GstPad* pad, std::string name);

void printPipelineState(GstPipeline* pipeline);

void printElementState(GstElement* elem);

void flushAndPrintBus(GstBus* bus, GstPipeline* pipe);

僅顯示PrintPipelineState和flushAndPrintBus的實現：

void printPipelineState(GstPipeline* pipeline){
    GstStateChangeReturn ret;
    GstState state;
    GstState pending;

    ret = gst_element_get_state(GST_ELEMENT(pipeline), &state, &pending, GST_CLOCK_TIME_NONE);
    if(ret == GST_STATE_CHANGE_FAILURE){
        std::cout << "failed querying state" << std::endl;
        return;
    }
    if(ret == GST_STATE_CHANGE_SUCCESS){
        std::cout << "successfully queried the state of pipeline" << std::endl;
        std::string str = "pipeline ";
        str += std::string(gst_element_state_get_name(state));
        std::cout << str << std::endl;
    }
}

void flushAndPrintBus(GstBus* bus, GstPipeline* pipe){
    if(!bus){
        return;
    }
    GstMessage* gm = gst_bus_pop(bus);
    while(gm){
        GError *err;
        gchar *debug_info;

        switch (GST_MESSAGE_TYPE (gm)) {
            case GST_MESSAGE_ERROR:
                gst_message_parse_error (gm, &err, &debug_info);
                 std::cout << "## BUS ## Error received from element " << GST_OBJECT_NAME (gm->src) <<  " : " <<  err->message << std::endl;;
                 std::cout << "## BUS ## Debugging information: " << (debug_info ? debug_info : "none")<< std::endl;
                 g_clear_error (&err);
                 g_free (debug_info);
                 break;
            case GST_MESSAGE_EOS:
                 std::cout << "## BUS ## End-Of-Stream reached." << std::endl;
                 break;
            case GST_MESSAGE_STATE_CHANGED:
            /* We are only interested in state-changed messages from the pipeline */
                if (GST_MESSAGE_SRC (gm) == GST_OBJECT (pipe)) {
                    GstState old_state, new_state, pending_state;
                    gst_message_parse_state_changed (gm, &old_state, &new_state, &pending_state);
                    std::cout << "## BUS ## Pipeline state changed from " <<
                        gst_element_state_get_name (old_state) << " to " << gst_element_state_get_name (new_state) << std::endl;
                }
                break;
            case GST_MESSAGE_STREAM_STATUS:
                GstStreamStatusType type;
                GstElement *owner;
                gst_message_parse_stream_status(gm, &type, &owner);
                std::cout << "Stream status : " << type << std::endl;
                break;
            default:
                /* We should not reach here */
                std::cout <<  "## BUS ## Unexpected message received: " << std::to_string(GST_MESSAGE_TYPE(gm)) << std::endl;

                break;
              }
        gm = gst_bus_pop(bus);
    }
}

現在我們在playerpipeline.h和playerpipeline.cpp中實現了管道：

#ifndef PLAYERPIPELINE_H
#define PLAYERPIPELINE_H

#include <QObject>
#include <glib.h>
#include <gst/gst.h>
#include <gst/video/videooverlay.h>
#include "gstmanips.h"
#include <vector>
#include <QTimer>
#include <QWidget>


class PlayerPipeline : public QObject
{
    Q_OBJECT
public:
    explicit PlayerPipeline(QString file, QWidget* display, QObject *parent = nullptr);
    void triggerNoMorePadSignal();
    void increasePreroll();
    void decreasePreroll();


signals:
    void noMorePadSignal();
    void decoderPrerolledSignal();
    void decoderBuiltSignal();
    void pipelineBuiltSignal();


public slots:
    void onNoMorePad();
    void onBusRefresh();
    void onDecoderPrerolled();
    void onPipelineBuilt();
    void play();
    void pause();

private:
    void changePipelineState(GstState state);
    void buildAndLinkConverters();
    void buildAndLinkMixer();
    void buildAndLinkPlaySink();

public:
    gint prerollCounter;
    bool prerolled;

private:
    //misc
    QString file;

    //pipeline
    GstBus* bus;
    GstPipeline* pipeline;
    QTimer* timer;
    unsigned timerStop;

    //decoder
    GstElement* decoder;
    std::vector<GstPad*> decoderAudioSrcs;
    GstPad* decoderVideoSrc;


    //audio convert (TODO add volume after)
    std::vector<GstElement*> audioConverters;
    std::vector<GstPad*> audioConvertersSrc;
    std::vector<GstPad*> audioConvertersSink;
    std::vector<GstElement*> audioQueues;

    //mixer
    GstElement* audioMixer;
    std::vector<GstPad*> audioMixerSinks;
    GstPad* audioMixerSrc;

    //sink
    GstElement* videoQueue;
    GstElement* playSink;
    GstPad* playSinkVideoSink;
    GstPad* playSinkAudioSink;
    GstElement* xvideo;
    QWidget* display;
};

#endif // PLAYERPIPELINE_H

#include "playerpipeline.h"
#include <assert.h>
#include <iostream>

void pp_no_more_pads_cb(GstElement* self, gpointer udata){
    (void)self;
    PlayerPipeline* db = (PlayerPipeline*)udata;
    std::cout << "no more pad :" <<std::endl;
    db->triggerNoMorePadSignal();
}

PlayerPipeline::PlayerPipeline(QString file, QWidget* display, QObject *parent)
    : QObject{parent}, prerollCounter(0), prerolled(false), file(file), display(display)
{
    pipeline = (GstPipeline*)gst_pipeline_new("PlayerPipeline");
    assert(pipeline);
    bus = gst_pipeline_get_bus(pipeline);
    assert(bus);
    decoder = gst_element_factory_make("uridecodebin", "decoder");
    assert(decoder);
    assert(gst_bin_add(GST_BIN(pipeline), decoder));
    assert(gst_element_sync_state_with_parent(decoder));
    std::string uri = std::string("file://")+file.toStdString();
    g_object_set(decoder, "uri", uri.c_str(), nullptr);

    QObject::connect(this, &PlayerPipeline::noMorePadSignal, this, &PlayerPipeline::onNoMorePad, Qt::QueuedConnection);
    g_signal_connect(decoder, "no-more-pads", G_CALLBACK(pp_no_more_pads_cb), this);
    QObject::connect(this, &PlayerPipeline::decoderPrerolledSignal, this, &PlayerPipeline::onDecoderPrerolled, Qt::QueuedConnection);
    QObject::connect(this, &PlayerPipeline::pipelineBuiltSignal, this, &PlayerPipeline::onPipelineBuilt);
    changePipelineState(GST_STATE_PLAYING);

    timer = new QTimer(this);
    QObject::connect(timer, &QTimer::timeout, this, &PlayerPipeline::onBusRefresh);
    timer->start(1);
    timerStop = 0;
}


void PlayerPipeline::triggerNoMorePadSignal()
{
    emit noMorePadSignal();
}


void PlayerPipeline::increasePreroll()
{
    g_atomic_int_inc(&prerollCounter);
}

void PlayerPipeline::decreasePreroll()
{
    if(g_atomic_int_dec_and_test(&prerollCounter)){
        this->prerolled = true;
        emit decoderPrerolledSignal();
    }
}

// FIXME there exists probably a better way to implem this
bool pp_caps_is_audio(GstCaps* caps){
    std::string description(gst_caps_to_string(caps));
    return description.rfind("audio/", 0) == 0;
}

// FIXME there exists probably a better way to implem this
bool pp_caps_is_video(GstCaps* caps){
    std::string description(gst_caps_to_string(caps));
    return description.rfind("video/", 0) == 0;
}

// FIXME maybe we need to use it to trigger when the pipeline is actually prerolled ?
GstPadProbeReturn blocked_cb(GstPad* pad, GstPadProbeInfo* info, gpointer udata){
    (void)pad;
    (void)info;
    std::cout << "blocked cb"<< std::endl;
    PlayerPipeline* pp = (PlayerPipeline*)udata;
    if(pp->prerolled){
        std::cout << "removing blocking probe" << std::endl;
        return GST_PAD_PROBE_REMOVE;
    } else {
    std::cout << "blocking probe installed !" << std::endl;
    pp->decreasePreroll();
    std::cout << "preroll decreased, exiting blocking_cb" << std::endl;
    return GST_PAD_PROBE_OK;
    }
}

void PlayerPipeline::onNoMorePad(){
    std::cout << "### LISTING PADS ###" << std::endl;
    changePipelineState(GST_STATE_PAUSED);
    GstIterator *it = gst_element_iterate_src_pads(decoder);
    GValue padV = G_VALUE_INIT;
    while(gst_iterator_next(it, &padV) == GST_ITERATOR_OK){
        std::cout << "PAD : " << std::endl;
        GstPad* pad = (GstPad*)g_value_get_object(&padV);
        assert(pad);
        GstCaps* caps = gst_pad_get_current_caps(pad);
        assert(caps);
        std::string description(gst_caps_to_string(caps));
        std::cout << description << std::endl;
        //FIXME some code duplication here, could likely be factorized ?
        if(pp_caps_is_audio(caps)){
            decoderAudioSrcs.push_back(pad);
        } else if (pp_caps_is_video(caps)){

            decoderVideoSrc = pad;
        } else {
            std::cout << "UNKNOWN PAD FOUND" << std::endl;
        }
        //probing the pad.
        this->increasePreroll();
        gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_BLOCK_DOWNSTREAM, blocked_cb, (gpointer)this, NULL);
    }
    std::cout << "waiting for pipeline to be prerolled" << std::endl;
}



void PlayerPipeline::onBusRefresh()
{
    //std::cout << "### timer timeouted ###" << std::endl;
    flushAndPrintBus(bus, pipeline);
    //timerStop += 1;
    //if (timerStop > 130){
    //    timer->stop();
    //}
}


void PlayerPipeline::onDecoderPrerolled()
{
    //here we can build the rest of the pipeline
    std::cout << "printing pipeline state after prerolling" << std::endl;
    printPipelineState(pipeline);
    buildAndLinkConverters();
    buildAndLinkMixer();
    buildAndLinkPlaySink();
    GST_DEBUG_BIN_TO_DOT_FILE(GST_BIN(pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline-prerolled");
    emit pipelineBuiltSignal();
}

void PlayerPipeline::buildAndLinkConverters(){
    std::cout << "BUILDING AND LINKING CONVERTERS" << std::endl;
    unsigned pRank = 0;
    for(auto dpad : decoderAudioSrcs){
        std::cout << "building audio queue" << std::endl;
        GstElement* queue = gst_element_factory_make("queue", (std::string("audioqueue_") + std::to_string(pRank)).c_str());
        audioQueues.push_back(queue);
        assert(queue);
        assert(gst_bin_add(GST_BIN(pipeline), queue));
        assert(gst_element_sync_state_with_parent(queue));
        GstPad* qsink = gst_element_get_static_pad(queue, "sink");
        GstPad* qsrc = gst_element_get_static_pad(queue, "src");
        assert(gst_pad_link(dpad, qsink) == GST_PAD_LINK_OK);
        std::cout << "building audio converter" << std::endl;
        GstElement* convert = gst_element_factory_make("audioconvert", (std::string("converter_") + std::to_string(pRank)).c_str());
        assert(convert);
        assert(gst_bin_add(GST_BIN(pipeline), convert));
        assert(gst_element_sync_state_with_parent(convert));
        GstPad* convsink = gst_element_get_static_pad(convert, "sink");
        assert(convsink);
        assert(gst_pad_link(qsrc, convsink) == GST_PAD_LINK_OK);
        audioConverters.push_back(convert);
        audioConvertersSink.push_back(convsink);
        //GstPad* convsrc = gst_element_get_static_pad(convert, "src");
        //assert(convsrc);
        //audioConvertersSrc.push_back(convsrc);
        pRank++;
    }
    std::cout << "printing pipeline state after converters" << std::endl;
    printPipelineState(pipeline);
   std::cout << "CONVERTERS : DONE" << std::endl;
}

void PlayerPipeline::buildAndLinkMixer(){
    std::cout << "BUILDING AND LINKING MIXER" << std::endl;
    audioMixer = gst_element_factory_make("audiomixer", "audioMixer");
    assert(audioMixer);
    assert(gst_bin_add(GST_BIN(pipeline), audioMixer));
    assert(gst_element_sync_state_with_parent(audioMixer));
    for(auto converter : audioConverters){
        assert(gst_element_link(converter, audioMixer));
    }
    std::cout << "printing pipeline state after adding mixer" << std::endl;
    printPipelineState(pipeline);
    std::cout << "MIXER : DONE" << std::endl;
}


GstPad* pp_requestPad(GstElement* elem, const char* tname, const char* name){
    GstPadTemplate* ptempl = gst_element_get_pad_template(elem, tname);
    if(ptempl == nullptr){
        std::cout << "failed to retreive pad template " << std::string(tname) << std::endl;
    }
    GstPad* pad = gst_element_request_pad(elem, ptempl, name, nullptr);
    if(pad == nullptr){
        std::cout << "pad request for " << std::string(name) << " returned nullptr" << std::endl;
    }
    return pad;
}

void PlayerPipeline::buildAndLinkPlaySink(){
    std::cout << "BUILDING AND LINKING PLAYSINK" << std::endl;
    playSink = gst_element_factory_make("playsink", "playSink");
    assert(playSink);
    WId xwinid = display->winId();
    gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(playSink), xwinid);
    assert(gst_bin_add(GST_BIN(pipeline), playSink));
    assert(gst_element_sync_state_with_parent(playSink));
    assert(gst_element_link(audioMixer, playSink));
    playSinkVideoSink = pp_requestPad(playSink, "video_raw_sink", "video_raw_sink");
    assert(decoderVideoSrc);
    assert(playSinkVideoSink);


    std::cout << "building video queue" << std::endl;
    GstElement* queue = gst_element_factory_make("queue", "videoqueue");
    videoQueue = queue;
    assert(queue);
    assert(gst_bin_add(GST_BIN(pipeline), queue));
    assert(gst_element_sync_state_with_parent(queue));
    GstPad* qsink = gst_element_get_static_pad(queue, "sink");
    assert(qsink);
    GstPad* qsrc = gst_element_get_static_pad(queue, "src");
    assert(qsrc);
    assert(gst_pad_link(decoderVideoSrc, qsink) == GST_PAD_LINK_OK);
    assert(gst_pad_link(qsrc, playSinkVideoSink) == GST_PAD_LINK_OK);

    GST_DEBUG_BIN_TO_DOT_FILE(GST_BIN(pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline-psink");
    std::cout << "PLAYSINK : DONE" << std::endl;
}

void PlayerPipeline::onPipelineBuilt(){
    //TODO implement here
    //here we can remove blocking probes. (by playing ?)
    std::cout << "UNPREROLLING THE PIPELINE" << std::endl;
    //printPipelineState(pipeline);
   // std::cout << "printing pipeline state before seeking" << std::endl;
   // printPipelineState(pipeline);
    assert(gst_element_seek(GST_ELEMENT(pipeline), 1.0,
                     GST_FORMAT_TIME,
                     GstSeekFlags(GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE),
                     GST_SEEK_TYPE_SET, 0,
                     GST_SEEK_TYPE_NONE, 0));
    // play();
    std::cout << "DONE" << std::endl;
}

void PlayerPipeline::play()
{
    GST_DEBUG_BIN_TO_DOT_FILE(GST_BIN(pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline-play");

}

void PlayerPipeline::pause()
{
    std::cout << "paused ..." << std::endl;
    changePipelineState(GST_STATE_PAUSED);
}


void PlayerPipeline::changePipelineState(GstState state)
{
    GstStateChangeReturn ret = gst_element_set_state(GST_ELEMENT(pipeline), state);
    if(ret == GST_STATE_CHANGE_FAILURE){
        std::cout << "could not set pipeline to state : " << gst_element_state_get_name (state) << std::endl;
        exit(124);
    }
}

方法和函數應按以下順序執行：

• PlayerPipeline::PlayerPipeline
• pp_no_more_pad_cb
• triggerNoMorePadSignal然后onNoMorePad
• 幾次（每個音軌一個）： blocked_cb然后decreasePreroll
• OnDecoderPrerolled ，按順序調用：
  • ```buildAndLinkConverters``
  • buildAndLinkMixer
  • buildAndLinkPlaySink
• onPipelineBuilt

並行（不完全並行，Qt 事件循環中的事件）每 1 毫秒調用一次onBusRefresh 。 它使用以下實現：

如果此時程序沒有卡在printPipelineState調用中，而我正在調用play()或pause()方法，則什么也不會發生（方法的 std output 除外）。 另外，我檢查過，此時墊仍然被阻塞。

編輯：調用play方法時的管道看起來像（我建議在新選項卡中打開以進行縮放）：

Answer 1

最后我找到了解決方案，即手動刪除探針，而不是依賴刷新搜索和回調黑魔法。

所以， add probe 返回一個gulong ：

gulong
gst_pad_add_probe (GstPad * pad,
                   GstPadProbeType mask,
                   GstPadProbeCallback callback,
                   gpointer user_data,
                   GDestroyNotify destroy_data)

所以我將它存儲在焊盤旁邊的向量中，然后使用以下方法手動刪除onPipelineBuilt中的探針：

gst_pad_remove_probe (GstPad * pad,
                      gulong id)

我不知道為什么回調+刷新沒有工作，盡管在本教程中顯示： https://gstreamer.freedesktop.org/documentation/application-development/advanced/pipeline-manipulation.html?gi-language= c我可能做錯了什么。

編輯：我很快就做了答案，如果您想要更多詳細信息或工作管道的完整代碼，請隨時詢問（通過評論）。