Obtaining frames from IP Camera with low latency

Question

I am currently using this command to get frames from my RTSP stream and reading frames from stdout:

ffmpeg -nostdin -rtsp_transport tcp -i <rtsp_stream> -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo -

However, I would like to get the same latency as when I see it via ffplay:

ffplay -fflags nobuffer -flags low_delay -tune zerolatency -framedrop -rtsp_transport tcp <rtsp_stream>

or when I play it via VLC Media > Open Network Stream with .network_caching=300ms.

I would like to know what other parameters I can use with my ffmpeg command to get an equivalent (or better) result compared to the ffplay command.

I have made references from: How to dump raw RTSP stream to file? , Open CV RTSP camera buffer lag , How to pipe output from ffmpeg using python? , bad ffmpeg performace compared to ffplay and VLC , How to minimize the delay in a live streaming with ffmpeg

My current implmentation:

FFMPEG_CMD = "ffmpeg -nostdin -rtsp_transport tcp -i <rtsp_stream> -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo -".split(" ")
WIDTH = 2560
HEIGHT = 1440

process = subprocess.Popen(FFMPEG_CMD, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

while True:
    raw_frame = process.stdout.read(WIDTH*HEIGHT*3)
    frame = np.frombuffer(raw_frame, np.uint8) 
    frame = frame.reshape((HEIGHT, WIDTH, 3))

    <do stuff with frame/ show frame etc.>

Thanks for reading.

---

ffmpeg command I am now using for < 1s latency.

ffmpeg -nostdin -flags low_delay -rtsp_transport tcp -i <rtsp_stream> -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo -

---

Implementation with suggestion(s) from Answers:

import subprocess
import numpy as np

FFMPEG_CMD = "ffmpeg -nostdin -flags low_delay -rtsp_transport tcp -i <rtsp_stream> -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo -".split(" ")
WIDTH = 2560
HEIGHT = 1440

process = subprocess.Popen(FFMPEG_CMD, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

raw_frame = np.empty((HEIGHT, WIDTH, 3), np.uint8) 
frame_bytes = memoryview(raw_frame).cast("B")

while process.poll() is None:
    process.stdout.readinto(frame_bytes)
    frame = raw_frame.reshape((HEIGHT, WIDTH, 3))

    <do stuff with frame/ show frame etc.>

Answer 1

I did some research about reducing the video latency.
My following answer demonstrates that the relevant FFmpeg flags are -probesize 32 and -flags low_delay .

The above flags are relevant for the video decoder side (receiver side).

The video encoding parameters "transmitter / encoder side" is more significant for determining the end to end latency.
Adding the argument -tune zerolatency reduces the encoder latency to minimum, but the required bandwidth is much higher (and probably not relevant for streaming over the inte.net).
I am going to restrict my answer to decoding latency, because it seems more relevant to the topic of your question.

The subject regarding "know how others obtain video frames with low latency" is a subject for a separate question (and I don't know the answer).

---

For comparing the latency differences between FFplay and FFmpeg (decoder), I created a "self contained" test sample.

Main "principles":

• Execute FFmpeg sub-process for streaming two RTSP output streams in parallel.
  The streamed video is synthetic pattern with frame counter as text over the video.
  The two output streams applies the same encoding parameters (only the port is different).
  The RTSP IP address is 127.0.0.1 (localhost).
  (Note: We may use tee muxer instead of encoding twice, but I never tried it).
• Execute FFplay sub-process that to decode and display one video stream.
• Execute FFmpeg sub-process that to decode the other video stream.
  OpenCV imshow is used for displaying the video.
• The displayed video with the larger counter is the one with the lower latency.

---

Code sample (updated):

import cv2
import numpy as np
import subprocess as sp
import shlex


rtsp_stream0 = 'rtsp://127.0.0.1:21415/live.stream'  # Use localhost for testing 
rtsp_stream1 = 'rtsp://127.0.0.1:31415/live.stream'
width = 256  # Use low resolution (for testing).
height = 144
fps = 30

# https://stackoverflow.com/questions/60462840/ffmpeg-delay-in-decoding-h264
ffmpeg_cmd = shlex.split(f'ffmpeg -nostdin -probesize 32 -flags low_delay -fflags nobuffer -rtsp_flags listen -rtsp_transport tcp -stimeout 1000000 -an -i {rtsp_stream0} -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo pipe:')


# FFplay command before updating the code (latency is still too high):  
# ffplay_cmd = shlex.split(f'ffplay -probesize 32 -analyzeduration 0 -sync ext -fflags nobuffer -flags low_delay -avioflags direct -rtsp_flags listen -strict experimental -framedrop -rtsp_transport tcp -listen_timeout 1000000 {rtsp_stream1}')

# Updated FFplay command - adding "-vf setpts=0" (fixing the latency issue):
# https://stackoverflow.com/questions/16658873/how-to-minimize-the-delay-in-a-live-streaming-with-ffmpeg
ffplay_cmd = shlex.split(f'ffplay -probesize 32 -analyzeduration 0 -sync ext -fflags nobuffer -flags low_delay -avioflags direct -rtsp_flags listen -strict experimental -framedrop -vf setpts=0 -rtsp_transport tcp -listen_timeout 1000000 {rtsp_stream1}')

# Execute FFplay to used as reference
ffplay_process = sp.Popen(ffplay_cmd)

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
process = sp.Popen(ffmpeg_cmd, stdout=sp.PIPE) #,stderr=sp.DEVNULL


# The following FFmpeg sub-process stream RTSP video.
# The video is synthetic video with frame counter (that counts every frame) at 30fps.
# The arguments of the encoder are almost default arguments - not tuned for low latency.
# drawtext filter with the n or frame_num function https://stackoverflow.com/questions/15364861/frame-number-overlay-with-ffmpeg
rtsp_streaming_process = sp.Popen(shlex.split(f'ffmpeg -re -f lavfi -i testsrc=size={width}x{height}:rate={fps} '
                                               '-filter_complex "drawtext=fontfile=Arial.ttf: text=''%{frame_num}'': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=72: box=1: boxcolor=white: boxborderw=5",'
                                               'split[v0][v1] '  # Split the input into [v0] and [v1]
                                               '-vcodec libx264 -pix_fmt yuv420p -g 30 -rtsp_transport tcp -f rtsp -muxdelay 0.1 -bsf:v dump_extra '
                                              f'-map "[v0]" -an {rtsp_stream0} '
                                               '-vcodec libx264 -pix_fmt yuv420p -g 30 -rtsp_transport tcp -f rtsp -muxdelay 0.1 -bsf:v dump_extra '
                                              f'-map "[v1]" -an {rtsp_stream1}'))


while True:
    raw_frame = process.stdout.read(width*height*3)

    if len(raw_frame) != (width*height*3):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Transform the byte read into a numpy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8)
    frame = frame.reshape((height, width, 3))

    # Show frame for testing
    cv2.imshow('frame', frame)
    key = cv2.waitKey(1)

    if key == 27:
        break
  
process.stdout.close()
process.wait()
ffplay_process.kill()
rtsp_streaming_process.kill()
cv2.destroyAllWindows()

---

Sample output before adding adding -vf setpts=0 :

Sample output (left side is OpenCV and right side is FFplay):

It looks like FFmpeg-OpenCV latency is lower by 6 frames before adding -vf setpts=0 to FFplay command.

Note: It took me some time to find the solution, and I decided to keep the result of the original post for showing the importance of adding the setpts filter.

---

Update:

Adding -vf setpts=0 solved the latency issue.

The latest answer from the following post suggests to add setpts video filter that resets all the video timestamps to zero.
It may not be a good idea with the present of audio stream, but when lowest video latency is required, this is the best solution I could find.

After adding -vf setpts=0 the latency of FFplay and OpenCV is about the same:

---

Repeating the test with mpv media player :

(Note: It seemed more relevant before I found the FFplay solution).

When applying all the mpv "latency hacks" from this page , the latency of mpv and OpenCV is about the same:

There must be a solution with FFplay, but I can't find it...

---

Code sample (using mpv instead of FFplay):

import cv2
import numpy as np
import subprocess as sp
import shlex

rtsp_stream0 = 'rtsp://127.0.0.1:21415/live.stream'  # Use localhost for testing 
rtsp_stream1 = 'rtsp://127.0.0.1:31415/live.stream'
width = 256  # Use low resolution (for testing).
height = 144
fps = 30

# https://stackoverflow.com/questions/60462840/ffmpeg-delay-in-decoding-h264
ffmpeg_cmd = shlex.split(f'ffmpeg -nostdin -probesize 32 -flags low_delay -fflags nobuffer -rtsp_flags listen -rtsp_transport tcp -stimeout 1000000 -an -i {rtsp_stream0} -pix_fmt bgr24 -an -vcodec rawvideo -f rawvideo pipe:')

# https://stackoverflow.com/questions/16658873/how-to-minimize-the-delay-in-a-live-streaming-with-ffmpeg
#ffplay_cmd = shlex.split(f'ffplay -probesize 32 -analyzeduration 0 -sync ext -fflags nobuffer -flags low_delay -avioflags direct -rtsp_flags listen -strict experimental -framedrop -rtsp_transport tcp -listen_timeout 1000000 {rtsp_stream1}')

# https://github.com/mpv-player/mpv/issues/4213
mpv_cmd = shlex.split(f'mpv --demuxer-lavf-o=rtsp_flags=listen --rtsp-transport=tcp --profile=low-latency --no-cache --untimed --no-demuxer-thread --vd-lavc-threads=1 {rtsp_stream1}')

# Execute FFplay to used as reference
#ffplay_process = sp.Popen(ffplay_cmd)

# Execute mpv media player (as reference)
mpv_process = sp.Popen(mpv_cmd)

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
process = sp.Popen(ffmpeg_cmd, stdout=sp.PIPE) #,stderr=sp.DEVNULL


# The following FFmpeg sub-process stream RTSP video.
# The video is synthetic video with frame counter (that counts every frame) at 30fps.
# The arguments of the encoder are almost default arguments - not tuned for low latency.
# drawtext filter with the n or frame_num function https://stackoverflow.com/questions/15364861/frame-number-overlay-with-ffmpeg
rtsp_streaming_process = sp.Popen(shlex.split(f'ffmpeg -re -f lavfi -i testsrc=size={width}x{height}:rate={fps} '
                                               '-filter_complex "drawtext=fontfile=Arial.ttf: text=''%{frame_num}'': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=72: box=1: boxcolor=white: boxborderw=5",'
                                               'split[v0][v1] '  # Split the input into [v0] and [v1]
                                               '-vcodec libx264 -pix_fmt yuv420p -g 30 -rtsp_transport tcp -f rtsp -muxdelay 0.1 -bsf:v dump_extra '
                                              f'-map "[v0]" -an {rtsp_stream0} '
                                               '-vcodec libx264 -pix_fmt yuv420p -g 30 -rtsp_transport tcp -f rtsp -muxdelay 0.1 -bsf:v dump_extra '
                                              f'-map "[v1]" -an {rtsp_stream1}'))


while True:
    raw_frame = process.stdout.read(width*height*3)

    if len(raw_frame) != (width*height*3):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Transform the byte read into a numpy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8)
    frame = frame.reshape((height, width, 3))

    # Show frame for testing
    cv2.imshow('frame', frame)
    key = cv2.waitKey(1)

    if key == 27:
        break
  
process.stdout.close()
process.wait()
#ffplay_process.kill()
mpv_process.kill()
rtsp_streaming_process.kill()
cv2.destroyAllWindows()

Answer 2

Assuming that the bottleneck is indeed somewhere in your example code (and not in <do stuff with frame/ show frame etc.> ), you can try to update numpy array as opposed to creating one every time:

frame = np.empty((HEIGHT, WIDTH, 3), np.uint8) 
frame_bytes = memoryview(frame).cast("b")
while True:
    process.stdout.readinto(frame_bytes) # fills the buffer of frame
    ...