OpenCV 读取具有多个流/轨道的视频文件

Question

I have a video file that contains multiple streams as shown below using VLC media player: Video Information

When I try to read it using Python + OpenCV using the following code:

vidObj = cv2.VideoCapture("video.avi")
ret, frame = vidObj.read()

I can only read the first track of the video. How can I read all the video tracks at the same time?

Answer 1

As far as I could tell, OpenCV does not allow choosing video stream, so this is not possible. However, you can do it rather easily with ffmpeg command line utilities:

import numpy as np
import json
import subprocess

def videoInfo(filename):
    proc = subprocess.run([
        *"ffprobe -v quiet -print_format json -show_format -show_streams".split(),
        filename
    ], capture_output=True)
    proc.check_returncode()
    return json.loads(proc.stdout)

def readVideo(filename):
    cmd = ["ffmpeg", "-i", filename]
    streams = 0
    for stream in videoInfo(filename)["streams"]:
        index = stream["index"]
        if stream["codec_type"] == "video":
            width = stream["width"]
            height = stream["height"]
            cmd += "-map", f"0:{index}"
            streams = streams + 1
    cmd += "-f", "rawvideo", "-pix_fmt", "rgb24", "-"
    shape = np.array([streams, height, width, 3])
    with subprocess.Popen(cmd, stdout=subprocess.PIPE) as proc:
        while True:
            data = proc.stdout.read(shape.prod())  # One byte per each element
            if not data:
                return
            yield np.frombuffer(data, dtype=np.uint8).reshape(shape)

Note that the code reads all video streams and assumes that each has the same resolution. It lacks proper error handling but got the job done in my scientific project.

For example, reading stereoscopic stream:

import matplotlib.pyplot as plt

for left, right in readVideo("testvideo.mkv"):
    plt.imshow(left)
    plt.show()
    plt.imshow(right)
    plt.show()