从两个视频源读取帧不同步 | OpenCV

Question

I am working on a python OpenCV script that will read from two video streams and put one onto the other one (Picture-In-Picture) and both are supposed to be in sync on the final video. My goal is to put one video on the bottom-left corner of the other one. It's somewhat working but the problem is that, even though I initialized both video sources and read frames from them at the same time, the videos are not in sync when I show the processed frames. I created a dedicated thread that reads frames from both videos at the same time and appends both (as a tuple) to a list of frame pairs (buffer). Then the main thread gets the frame pair from the buffer, processes the final frame, and shows it using imshow . But one video is several seconds ahead of the other one. I tried reading frames in the main thread but the same problem. The program also reads some information from subtitle file and adds text to the final frame. But that is working fine. So just ignore the getTelemetry() function. I would really appreciate some help. Thanks

import cv2
import time
import _thread

telemetry = []
tel = []


def getTelemetry():
    global telemetry, tel
    file = open("subs.srt", "r")
    lines = file.readlines()
    chars = '-.0123456789'
    distance, altitude, horizontal_speed, vertical_speed = None, None, None, None
    for line in lines:
        if line.startswith("F"):
            # print(line.strip('\n'))
            distance = line[line.index('D ')+2:line.index(', H') - 1]
            altitude = line[line.index('H ')+2:line.index(', H.S') - 1]
            horizontal_speed = line[line.index(
                'H.S ')+4:line.index(', V.S') - 3]
            vertical_speed = line[line.index('V.S ')+4:-5]
            for char in distance:
                if char not in chars:
                    distance = '0'
            for char in altitude:
                if char not in chars:
                    altitude = '0'
            for char in horizontal_speed:
                if char not in chars:
                    horizontal_speed = '0'
            for char in vertical_speed:
                if char not in chars:
                    vertical_speed = '0'

            distance_float = float(distance)
            altitude_float = float(altitude)
            horizontal_speed_float = float(horizontal_speed)
            vertical_speed_float = float(vertical_speed)
            telemetry.append([distance_float, altitude_float,
                              horizontal_speed_float, vertical_speed_float])
            # print(distance_float, altitude_float, horizontal_speed_float,
            #      vertical_speed_float)

    # Put each element of telemetry 60 times in tel
    for i in range(0, len(telemetry)):
        for j in range(0, 60):
            tel.append(telemetry[i])
    # print(len(tel))


frame_buffer = []


def fil_buffer():
    global frame_buffer
    video = cv2.VideoCapture("Input_File.avi")
    video2 = cv2.VideoCapture("Screenrecorder-2021-12-07-16-15-26-217.mp4")
    # For each frame in the video
    while(video.isOpened() and video2.isOpened()):
        ret, frame = video.read()
        ret2, frame2 = video2.read()
        if ret == True and ret2 == True:
            frame_buffer.append((frame, frame2))

    video.release()
    video2.release()


if __name__ == '__main__':
    getTelemetry()
    _thread.start_new_thread(fil_buffer, ())
    time.sleep(5)
    print(len(frame_buffer))
    j = 0
    while len(frame_buffer) != 0:
        x = frame_buffer.pop(0)
        frame1 = x[0]
        frame2 = x[1]
        # print(tel[j][0])
        cv2.putText(frame1, "Distance: " + str(tel[j][0]), (10, 30),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
        cv2.putText(frame1, "Altitude: " + str(tel[j][1]), (10, 60),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
        cv2.putText(frame1, "Horizontal Speed: " + str(tel[j][2]), (10, 90),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
        cv2.putText(frame1, "Vertical Speed: " + str(tel[j][3]), (10, 120),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
        # Resize frame2 to 25%
        frame2 = cv2.resize(frame2, (0, 0), fx=0.25, fy=0.25)
        # Put frame2 over frame1
        frame1[500:frame2.shape[0]+500, 0:frame2.shape[1]] = frame2
        cv2.imshow('frame', frame1)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

Answer 1

As mentioned in the comments by Mark, the cameras need not be in be sync, to begin with. They can have different FPS even after having identical hardware. Also, in your fil_buffer() function, the reading of frames from both cameras is sequential, ie you read camera A first and then camera B. This will definitely increase some level of lag for camera A. I would like to approach the problem this way.

1. Run the frame reading task for each camera in a separate thread of their own.
2. In the same thread, you can push the read camera frame in their own STACK(not queue/list).
3. In your main function, you should read that stack, and clear it instantly. This makes sure you are always reading the latest available frame in the stack.

I am assuming that the processing time in your main function will be greater than the time taken to read the frame. This should hold true for your application.

(A few months back, I wrote this article about getting a higher FPS with cameras. Its implementation is in C++ but it has a structure to run multiple cameras in separate threads of their own. I have been using the same structure to read multiple FLIR cameras to read them at FPS of 50-55 and in sync with each other.)