为什么 OpenCV 读取视频比 FFMPEG 快？

Question

I noticed that OpenCV reads video frames almost 2x faster than FFMPEG.

Why is that? I thought all OpenCV does is call FFMPEG under the hood, possibly adding its own overhead.

Here's the code I use to obtain these results

import cv2
import time
import numpy as np

cap = cv2.VideoCapture("BigBuckBunny.mp4", apiPreference=cv2.CAP_FFMPEG)
frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

start = time.perf_counter()
while True:
    ret, frame = cap.read()
    if ret is False:
        break
    assert frame.shape == (720, 1280, 3)
    assert frame.dtype == np.uint8
end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
# Output: 692.3 frames per second

cap.release()

For FFMPEG (using the python-ffmpeg library:

import ffmpeg
import numpy as np
import time

vid_info = ffmpeg.probe("BigBuckBunny.mp4")['streams'][1]
frames = int(vid_info['nb_frames'])

process1 = (
    ffmpeg
    .input("BigBuckBunny.mp4")
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
)
print(process1.compile())
# Output: ['ffmpeg', '-i', 'BigBuckBunny.mp4', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:']


process1 = process1.run_async(pipe_stdout=True)

start = time.perf_counter()
while True:
    in_bytes = process1.stdout.read(1280 * 720 * 3)
    if not in_bytes:
        break
    frame = np.frombuffer(in_bytes, np.uint8).reshape([720, 1280, 3])
end = time.perf_counter()
print(f"{frames/(end-start):.1f} frames per second")
# Output: 373.6 frames per second
process1.wait()

Here's information about the video (~10 minutes length)

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BigBuckBunny.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isomavc1mp42
    creation_time   : 2010-01-10T08:29:06.000000Z
  Duration: 00:09:56.47, start: 0.000000, bitrate: 2119 kb/s
  Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1991 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]

And FFMPEG and OpenCV versions:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers

opencv-python-headless        4.6.0.66

Answer 1

It may be due to the fact how the libraries are implemented and the overhead of pipes.

1. cv2

cv2 implemented in c++ and uses libavcodec library (part of ffmpeg) for decoding frames:

https://github.com/opencv/opencv/blob/97c6ec6d49cb78321eafe6fa220ff80ebdc5e2f4/modules/videoio/src/cap_ffmpeg_impl.hpp#L1299

cv2 does not use ffmpeg as binary executable and uses the part of ffmpeg specifically designed for reading/decoding frames.

2. ffmpeg-python is a wrapper around ffmpeg executable (which in turn uses libavcodec under the hood too) that calls ffmpeg with subprocess and returns results through the pipes.

Answer 2

Good stuff. Interestingly, under Windows 10 & FFmpeg 5.0.1, my results are the reverse of yours (but much closer):

• OpenCV: 491.9 frames per second
• FFmpeg-Python: 519.4 frames per second

And out of curiosity, I tried my ffmpegio package to see how much overhead it incurs, it came out: 507.5 frames per second (code below) Both FFmpeg-Python and FFmpegIO scoring in the same ballpark indicates these numbers aren't fluke.

So, it's likely Win/Linux or 4.4/5.0 differences or both. I retested FFmpeg-Python with FFmpeg 4.4.1 and got: 428.1 frames per second (log reporting speed=17.9x down from x21.7 . I'd say FFmpeg devs optimized the color conversion codebase in v5 releases to match or surpass OpenCV. (Or the tested OpenCV version uses the libav libraries associated with FFmpeg v5)

test_ffmpegio.py

import ffmpegio
import time

frames = ffmpegio.probe.video_streams_basic("BigBuckBunny.mp4")[0]['nb_frames']

print(frames)

with ffmpegio.open("BigBuckBunny.mp4",'rv') as f:
    start = time.perf_counter()
    for frame in f:
        pass # frame is 1x1280x720x3 rgb np array
    end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")