Issue with configuration of cv2.VideoWriter and GStreamer

Question

I am having trouble setting up a GStreamer pipeline to forward a video stream over UDP via OpenCV. I have a laptop, and an AGX Xavier connected to the same network. The idea is to forward the webcam video feed to AGX which will do some OpenCV optical flow estimation on the GPU (in Python), draw flow vectors on the original image and send it back to my laptop. Up until now, I can configure two pipelines. As a minimum example, I have made two bash scripts and a Python script that ideally would function as pass-through over OpenCV's VideoCapture and VideoWriter objects.

servevideo.bash:

#!/bin/bash

gst-launch-1.0 v4l2src device=[device-fd] \
    ! video/x-raw, width=800, height=600, framerate=24/1 \
    ! jpegenc ! rtpjpegpay ! rtpstreampay \
    ! udpsink host=[destination-ip] port=12345

receivevideo.bash:

#!/bin/bash

gst-launch-1.0 -e udpsrc port=12344 \
    ! application/x-rtp-stream,encoding-name=JPEG \
    ! rtpstreamdepay ! rtpjpegdepay ! jpegdec \
    ! autovideosink

If I run these two scripts on either the same computer or on two different computers on the network, it works fine. When I throw my Python script (listed below) in the mix, I start to experience issues. Ideally, I would run the bash scripts on my laptop with the intended setup in mind while running the Python script on my Jetson. I would then expect to see the webcam video feed at my laptop after taking a detour around the Jetson.

webcam_passthrough.py:

#./usr/bin/python3.6

import cv2

video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)
video_out = cv2.VideoWriter("appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)

while True:
    ret, frame = video_in.read()

    if not ret: break

    video_out.write(frame)
    cv2.imshow('Original', frame)

    key = cv2.waitKey(1) & 0xff
    if key == 27: break

cv2.destroyAllWindows()
video_out.release()
video_in.release()

With the following Python script, I can visualise the frames via cv2.imshow received from the pipeline set up by the servevideo.bash script. So I think my problem is connected to how I am setting up the VideoWriter video_out in OpenCV. I have verified my two bash scripts are working when I am relaying the webcam video feed between those two pipelines created, and I have verified that the cv2.VideoCapture receives the frames. I am no expert here, and my GStreamer knowledge is almost non-existent, so there might be several misunderstandings in my minimum example. It would be greatly appreciated if some of you could point out what I am missing here.

I will also happily provide more information if something is unclear or missing.

EDIT: So it seems the intention of my minimum example was not clearly communicated.

The three scripts provided as a minimum example serve to relay my webcam video feed from my laptop to the Jetson AGX Xavier who then relays the video-feed back to the laptop. The servevideo.bash creates a GStreamer pipeline on the laptop that uses v4l2 to grab frames from the camera and relay it on to a UDP socket. The webcam_passthrough.py runs on the Jetson where it "connects" to the UDP socket created by the pipeline running on the laptop. The Python script serves a passthrough which ideally will open a new UDP socket on another port and relay the frames back to the laptop. The receivevideo.bash creates yet another pipeline on the laptop for receiving the frames that were passed through the Python script at the Jetson. The second pipeline on the laptop is only utilised for visualisation purpose. Ideally, this minimum example shows the "raw" video feed from the camera connected to the laptop.

The two bash scripts are working in isolation, both running locally on the laptop and running receivevideo.bash remotely on another computer.

The cv2.VideoCapture configuration in the Python script also seems to work as I can visualise the frames (with cv2.imshow ) received over the UDP socket provided by the servevideo.bash script. This is working locally and remotely as well. The part that is causing me some headache (I believe) is the configuration of cv2.VideoWriter ; ideally, that should open a UDP socket which I can "connect" to via my receivevideo.bash script. I have tested this locally and remotely but to no avail.

When I run receivevideo.bash to connect to the UDP socket provided by the Python script I get the following output:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

This does not seem wrong to me, I have tried to run the different scripts with GST_DEBUG=3 which gave some warnings, but as the pipeline configurations are basically the same in the bash scripts and for the cv2 VideoCapture and VideoWriter I do not add much value to those warnings. As an example I have included one such warning below:

0:00:06.595120595  8962      0x25b8cf0 WARN              rtpjpegpay gstrtpjpegpay.c:596:gst_rtp_jpeg_pay_read_sof:<rtpjpegpay0> warning: Invalid component

This warning is printed continuously running the Python script with GST_DEBUG=3 . Running the receivevideo.bash with the same debug level gave:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.013911480  9078 0x55be0899de80 FIXME           videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out:<jpegdec0> Sub-class should implement drain()
Setting pipeline to PLAYING ...
New clock: GstSystemClock

I hope my intention is clearer now, and as I already pointed out I believe something is wrong with my cv2.VideoWriter in the Python script, but I am no expert and GStreamer is far from something that I use every day. Thus, I may have misunderstood something.

EDIT 2: So now I have tried to split the two pipelines into two separate processes as suggested by @abysslover. I still see the same result, and I still have no clue why that is. My current implementation of the Python script is listed below.

webcam_passthrough.py:

#!/usr/bin/python3.6

import signal, cv2
from multiprocessing import Process, Pipe

is_running = True

def signal_handler(sig, frame):
    global is_running
    print("Program was interrupted - terminating ...")
    is_running = False

def produce(pipe):
    global is_running
    video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)

    while is_running:
        ret, frame = video_in.read()
        if not ret: break
        print("Receiving frame ...")

        pipe.send(frame)

    video_in.release()

if __name__ == "__main__":
    consumer_pipe, producer_pipe = Pipe()

    signal.signal(signal.SIGINT, signal_handler)
    producer = Process(target=produce, args=(producer_pipe,))

    video_out = cv2.VideoWriter("appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
    producer.start()

    while is_running:
        frame = consumer_pipe.recv()
        video_out.write(frame)
        print("Sending frame ...")

    video_out.release()
    producer.join()

The pipe that I have created between the two processes is providing a new frame as expected. When I try to listen to UDP port 12344 with netcat , I do not receive anything that is the same thing as before. I also have a hard time understanding how differentiating the pipelines are changing much as I would expect them to already run in different contexts. Still, I could be wrong concerning this assumption.

Answer 1

Note: I cannot write a comment due to the low reputation.

According to your problem description, it is difficult to understand what your problem is.

Simply, you will run two bash scripts ( servevideo.bash and receivevideo.bash ) on your laptop, which may receive and send web-cam frames from the laptop (?), while a Python script( webcam_passthrough.py ) runs on a Jetson AGX Xavier.

Your bash scripts work, so I guess you have some problems in the Python script. According to your explanation, you've already got the frames from the gst-launch in the bash scripts and visualized the frames.

Thus, what is your real problem? What are you trying to solve using the Python script?

The following statement is unclear to me.

When I throw my Python script (listed below) in the mix, I start to experience issues.

How about the following configuration?

servevideo.bash:

#!/bin/bash
gst-launch-1.0 videotestsrc device=[device-fd] \
    ! video/x-raw, width=800, height=600, framerate=20/1 \
    ! videoscale
    ! videoconvert
    ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast
    ! rtph264pay
    ! udpsink host=[destination-ip] port=12345

receivevideo.bash

#!/bin/bash
gst-launch-1.0 -v udpsrc port=12345 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" \
     ! rtph264depay \
     ! decodebin \
     ! videoconvert \
     ! autovideosink

Python script:

import numpy as np
import cv2
from multiprocessing import Process

def send_process():
    video_in = cv2.VideoCapture("videotestsrc ! video/x-raw,framerate=20/1 ! videoscale ! videoconvert ! appsink", cv2.CAP_GSTREAMER)
    video_out = cv2.VideoWriter("appsrc ! videoconvert ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay ! udpsink host=[destination_ip] port=12345", cv2.CAP_GSTREAMER, 0, 24, (800,600), True)

    if not video_in.isOpened() or not video_out.isOpened():
        print("VideoCapture or VideoWriter not opened")
        exit(0)

    while True:
        ret,frame = video_in.read()

        if not ret: break

        video_out.write(frame)

        cv2.imshow("send_process", frame)
        if cv2.waitKey(1)&0xFF == ord("q"):
            break

    video_in.release()
    video_out.release()

def receive_process():
    cap_receive = cv2.VideoCapture('udpsrc port=12345 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! appsink', cv2.CAP_GSTREAMER)

    if not cap_receive.isOpened():
        print("VideoCapture not opened")
        exit(0)

    while True:
        ret,frame = cap_receive.read()

        if not ret: break

        cv2.imshow('receive_process', frame)
        if cv2.waitKey(1)&0xFF == ord('q'):
            break

    cap_receive.release()

if __name__ == '__main__':
    s = Process(target=send_process)
    r = Process(target=receive_process)
    s.start()
    r.start()
    s.join()
    r.join()

    cv2.destroyAllWindows()

I cannot test with codes since I do not have your configuration. I think that the receiver and sender needs to be forked into two separate processes using multiprocessing.Process in Python. You may need to adjust some detailed parameters in order to work with these scripts in your configuration.

Good luck to you.

Answer 2

you were very close to the solution. The problem lies in the warning you yourself noticed warning: Invalid component . The problem is that rtp jpeg payloader gets stuck due to not supporting video format it is getting. Check this

However I was blind and missed what you wrote and went full debug mode into the problem.

So lets just keep the debug how-to for others or for similar problems:

1, First debugging step - check with wireshark if the receiving machine is getting udp packets on port 12344. Nope it does not.

2, Would this work without opencv stuff? Lets check with replacing opencv logic with some random processing - say rotation of video. Also eliminate appsrc/appsink to simplify.

Then I used this:

GST_DEBUG=3 gst-launch-1.0 udpsrc port=12345, application/x-rtp-stream.encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! rotate angle=0.45 ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! queue ! udpsink host=[my ip] port=12344

Hm now I get weird warnings like:

0:00:00.174424533 90722 0x55cb38841060 WARN              rtpjpegpay gstrtpjpegpay.c:596:gst_rtp_jpeg_pay_read_sof:<rtpjpegpay0> warning: Invalid component
WARNING: from element /GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0: Invalid component

3, Quick search yielded above mentioned GStreamer forum page.

4, When I added video/x-raw,format=I420 after videoconvert it started working and my second machine started getting the udp packets.

5, So the solution to your problem is just limit the jpegenc to specific video format that the subsequent rtp payloader can handle:

#!/usr/bin/python3

import signal, cv2
from multiprocessing import Process, Pipe

is_running = True

def signal_handler(sig, frame):
    global is_running
    print("Program was interrupted - terminating ...")
    is_running = False

def produce(pipe):
    global is_running
    video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)

    while is_running:
        ret, frame = video_in.read()
        if not ret: break
        print("Receiving frame ...")

        pipe.send(frame)

    video_in.release()

if __name__ == "__main__":
    consumer_pipe, producer_pipe = Pipe()

    signal.signal(signal.SIGINT, signal_handler)
    producer = Process(target=produce, args=(producer_pipe,))

    # the only edit is here, added video/x-raw capsfilter:     <-------
    video_out = cv2.VideoWriter("appsrc ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[receiver ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
    producer.start()

    while is_running:
        frame = consumer_pipe.recv()
        rr = video_out.write(frame)
        print("Sending frame ...")
        print(rr)

    video_out.release()
    producer.join()