How to extract frames from a .yuv video file (YUV420) using python and openCV?

Question

I need to read a yuv video file, extract individual frames from it, convert it to grayscale and then calculate Lucas Kanade optical flow between adjacent frames. I was initially using mp4 videos and this was my code for extracting individual frames:

import cv2 as cv
import numpy as np
cap = cv.VideoCapture('C:\\Users\\Ann Baiju\\Project\\video_tampering_dataset\\videos\\h264_lossless\\07_forged.mp4')

ret, frame1 = cap.read()
prvs = cv.cvtColor(frame1, cv.COLOR_BGR2GRAY)

height, width, _ = frame1.shape

while(1):
    ret, frame2 = cap.read()
    if ret==False: break
    next = cv.cvtColor(frame2, cv.COLOR_BGR2GRAY)
    #Code for calculating Lucas Kanade optical flow
    N=N+1
    prvs = next

cap.release()

Now some things changed and I have to use a dataset of yuv video files. But when I give a yuv file to VideoCapture() I'm getting error as follows:

[IMGUTILS @ 00000078a4bee5c0] Picture size 0x0 is invalid [ERROR:0] global C:\\projects\\opencv-python\\opencv\\modules\\videoio\\src\\cap.cpp (116) cv::VideoCapture::open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.1.2) C:\\projects\\opencv-python\\opencv\\modules\\videoio\\src\\cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): C:\\Users\\Ann Baiju\\Project\\Copy_Move_Datasets\\new original\\DERF\\hall.yuv in function 'cv::icvExtractPattern'

Traceback (most recent call last): File "test1.py", line 6, in prvs = cv.cvtColor(frame1, cv.COLOR_BGR2GRAY) cv2.error: OpenCV(4.1.2) C:\\projects\\opencv-python\\opencv\\modules\\imgproc\\src\\color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor'

How do I solve this? Also I understand yuv is a raw video file format and does not have size or fps information in it. Is there some way it can be inferred from the file or do I have to manually input that information?

Regarding my question as to how to get frame size information (height and width) from yuv video is it possible to convert the yuv video to some other format (say mp4) using FFmpeg, obtain the information from it and then delete the mp4 video and continue to work with the yuv video? If so how to do that?

import cv2
import numpy as np
import os
import subprocess as sp

yuv_filename = 'can_0.yuv'
#flow=[]

width, height = 320, 240

file_size = os.path.getsize(yuv_filename)
n_frames = file_size // (width*height*3 // 2)
f = open(yuv_filename, 'rb')


old_yuv = np.frombuffer(f.read(width*height*3//2), dtype=np.uint8).reshape((height*3//2, width))
cv2.imshow('frame',old_yuv)
cv2.waitKey(3000)

# Convert YUV420 to Grayscale
old_gray = cv2.cvtColor(old_yuv, cv2.COLOR_YUV2GRAY_I420)
cv2.imshow('frame_gs',old_gray)
cv2.waitKey(3000)

When I run the above code, the yuv image I get is:

Is that normal for a yuv image or some kind of resolution problem? Also why is there no color? When I convert it to grayscale however, it comes out normal:

The original frame is (as viewed using yuvplayer 2.5):

Answer 1

There is no single YUV420 file format, but many possible pixel ordering.

I created a "self contained" code sample that demonstrates reading YUV420 frames.
The code sample:

• Generates synthetic video in YUV420 format using FFmpeg .
  For executing the sample, download (or install) latest stable version.
  For Windows OS, you may place ffmpeg.exe in the same path as the Python script.
• Read the YUV420 frame by frame, convert to BGR, and display each frame.
  You don't need the color, but it may be important for testing.
• Convert YUV420 to Grayscale, and display each frame as Grayscale.

FFmpeg creates the YUV in I420 planar format:

YYYYYY
YYYYYY
YYYYYY
YYYYYY
UUU
UUU
VVV
VVV

The range of the YUV pixels is "Limited range":

• Y range is [16, 235].
• U and V range is [16, 240].

---

Here is the code:

import cv2
import numpy as np
import os
import subprocess as sp

# Build synthetic video and read binary data into memory (for testing):
#########################################################################
mp4_filename = 'input.mp4'  # the mp4 is used just as reference
yuv_filename = 'input.yuv'
width, height = 640, 480
fps = 1 # 1Hz (just for testing)

# Build synthetic video, for testing (the mp4 is used just as reference):
sp.run('ffmpeg -y -f lavfi -i testsrc=size={}x{}:rate=1 -vcodec libx264 -crf 18 -t 10 {}'.format(width, height, mp4_filename))
sp.run('ffmpeg -y -f lavfi -i testsrc=size={}x{}:rate=1 -pix_fmt yuv420p -t 10 {}'.format(width, height, yuv_filename))
#########################################################################


file_size = os.path.getsize(yuv_filename)

# Number of frames: in YUV420 frame size in bytes is width*height*1.5
n_frames = file_size // (width*height*3 // 2)

# Open 'input.yuv' a binary file.
f = open(yuv_filename, 'rb')

for i in range(n_frames):
    # Read Y, U and V color channels and reshape to height*1.5 x width numpy array
    yuv = np.frombuffer(f.read(width*height*3//2), dtype=np.uint8).reshape((height*3//2, width))

    # Convert YUV420 to BGR (for testing), applies BT.601 "Limited Range" conversion.
    bgr = cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR_I420)

    # Convert YUV420 to Grayscale
    gray = cv2.cvtColor(yuv, cv2.COLOR_YUV2GRAY_I420)

    #Show RGB image and Grayscale image for testing
    cv2.imshow('rgb', bgr)
    cv2.waitKey(500)  # Wait a 0.5 second (for testing)
    cv2.imshow('gray', gray)
    cv2.waitKey(500)  # Wait a 0.5 second (for testing)

f.close()

cv2.destroyAllWindows()

Note:

• There is a good chances that your YUV420 file is not in I420, "limited range" format.

---

Update:

Example for converting YUV420 to uncompressed AVI using FFmpeg :

• The input file must be with .yuv extension.

In command line console:

ffmpeg -y -s 640x480 -pixel_format yuv420p -i input.yuv -vcodec rawvideo -pix_fmt bgr24 input.avi

Using subprocess within Python:

sp.run('ffmpeg -y -s {}x{} -pixel_format yuv420p -i input.yuv -vcodec rawvideo -pix_fmt bgr24 input.avi'.format(width, height))

• The uncompressed AVI video file is going to be very large (twice the size of the YUV file).
  In case video quality is not most important, you may use H.264 compression (for example):

   ffmpeg -y -s 640x480 -pixel_format yuv420p -i input.yuv -vcodec libx264 -crf 17 -pix_fmt yuv420p input.mp4

  The -crf 17 argument keeps the quality high (almost lossless).
  Note: The file format .mp4 or .avi is not important when H.264 encoding is used (but .mp4 is more common when using H.264 encoding).