更改 GStreamer 视频设备分辨率

Question

I am trying to use a USB 3 capture card with GStreamer, however the only exposed resolution on the capture card seems to be 1080p60 and this seems to be too much data for my Coral Dev Board to handle, convert and process through object detection in a timely manner.

I have used a USB 2 card at 480p30 and this works but would like something a bit higher. I have tried two USB 3 cards, an Elgato game capture hd60 s+ and a pengo 1080p grabber, both of which seem to have the same issue.

When I use my USB 2 card, it exposes multiple different resolutions with different framerates, both in OBS on windows and when listing available formats on Linux, however, both USB3 cards only expose 1080p60.

I get very slow, laggy inference at 1080p60, and the program crashes with any other parameters, including 1080p30, 720p60, and 720p30. I think 720p30 would be ideal, however am unsure how to achieve this.

I have been using this script to run inference on a coral dev board 4GB. I would like to stick to python if possible.

Warning when 1080p60 is used, as well as being slow and laggy:

Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/glsvgoverlaysink+GlSvgOverlaySink:overlaysink/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink:

There may be a timestamping problem, or this computer is too slow

The gstreamer code:

v4l2src device=/dev/video1 ! video/x-raw,width=1920,height=1080,framerate=60/1 ! decodebin ! glupload ! tee name=t

t. ! queue ! glfilterbin filter=glbox name=glbox ! video/x-raw,format=RGB,width=300,height=300 ! appsink name=appsink emit-signals=true max-buffers=1 drop=true

t. ! queue ! glsvgoverlaysink name=overlaysink````

Answer 1

you can use cv2 ( pip install opencv-python ) to access the USB camera.

Here is a little example of how you can get the image from the Camera and show it in a separate Window

# import the opencv library
import cv2
  
  
# define a video capture object
vid = cv2.VideoCapture(0)
  
while(True):
      
    # Capture the video frame
    # by frame
    ret, frame = vid.read()
  
    # Display the resulting frame
    cv2.imshow('frame', frame)
      
    # the 'q' button is set as the
    # quitting button you may use any
    # desired button of your choice
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
  
# After the loop release the cap object
vid.release()
# Destroy all the windows
cv2.destroyAllWindows()

Answer 2

Your gstreamer pipeline appears to be inefficiently written.

I'd suggest to use videoscale and capsfilter at the earliest point unless you really want to use 1080p stream.

Eg,

v4l2src device=/dev/video1 ! videoconvert ! videoscale ! videorate ! video/x-raw,format=RGB,width=1080,height=720,framerate=30/1 ! # this will ensure you get 720@30 even if this camera doesn't support it.
# decodebin ! YOU DO NOT NEED decodebin for video/x-raw
glupload ! tee name=t

t. ! queue ! glfilterbin filter=glbox name=glbox ! video/x-raw,format=RGB,width=300,height=300 ! appsink name=appsink emit-signals=true max-buffers=1 drop=true

t. ! queue ! glsvgoverlaysink name=overlaysink````

Generally, such GStreamer off-the-shelf filters are faster and more efficient (lower memory) than python-cv (or even C++-CV).

Besides, for such queues with high-latency filters, consider limiting the queue size and making them leaky (I use leaky=2 for real-time inferences).

Another suggestion: you may use nnstreamer instead of cv filters if what you do is running tensorflow-lite models with edge-tpu unless CV interfaces provide higher efficiency than invoking tensorflow-lite delegations.