Can't write frames to a video with multiprocessing + cv2

Question

I have a code which breaks down a video into frames and edits the image and puts it back into a video, but I am realizing that it's really slow... So I looked into multiprocessing for speeding up the code, and it works, As I can see it processes the images much faster, but the problem is, when I add those frames to a new video, it doesn't work, the video remains empty!

Here is my code:

# Imports
import cv2, sys, time
import numpy as np

from scipy.ndimage import rotate
from PIL import Image, ImageDraw, ImageFont, ImageOps
import concurrent.futures
def function(fullimg):
    img = np.array(Image.fromarray(fullimg).crop((1700, 930, 1920-60, 1080-80)))
    inpaintRadius = 10
    inpaintMethod = cv2.INPAINT_TELEA
    textMask = cv2.imread('permanentmask.jpg', 0)
    final_result = cv2.inpaint(img.copy(), textMask, inpaintRadius, inpaintMethod)
    text = Image.fromarray(np.array([np.array(i) for i in final_result]).astype(np.uint8)).convert('RGBA')
    im = np.array([[tuple(x) for x in i] for i in np.zeros((70, 160, 4))])
    im[1:-1, 1:-1] = (170, 13, 5, 40)
    im[0, :] = (0,0,0,128)
    im[1:-1, [0, -1]] = (0,0,0,128)
    im[-1, :] = (0,0,0,128)
    im = Image.fromarray(im.astype(np.uint8))
    draw = ImageDraw.Draw(im)
    font = ImageFont.truetype('arialbd.ttf', 57)
    draw.text((5, 5),"TEXT",(255,255, 255, 128),font=font)
    text.paste(im, mask=im)
    text = np.array(text)
    fullimg = Image.fromarray(fullimg)
    fullimg.paste(Image.fromarray(text), (1700, 930, 1920-60, 1080-80))
    fullimg = cv2.cvtColor(np.array(fullimg), cv2.COLOR_BGR2RGB)
    return fullimg
cap = cv2.VideoCapture('before2.mp4')
_fourcc = cv2.VideoWriter_fourcc(*'MPEG')
out = cv2.VideoWriter('after.mp4', _fourcc, 29.97, (1280,720))
frames = []
lst = []
while cap.isOpened():
    ret, fullimg = cap.read()
    if not ret:
        break
    frames.append(fullimg)
    
    if len(frames) >= 8:
        if __name__ == '__main__':
            with concurrent.futures.ProcessPoolExecutor() as executor:
                results = executor.map(function, frames)
                for i in results:
                    print(type(i))
                    out.write(i)
        frames.clear() 
cap.release()
out.release()
cv2.destroyAllWindows() # destroy all opened windows

My code inpaints a watermark and adds another watermark using PIL.

If I don't use multiprocessing the code works. But if I do use multiprocessing , it gives an empty video.

Answer 1

I am not that familiar with OpenCV , but there seems to be a few things that should be corrected in your code. First, if you are running under Windows, as you appear to be because you have if __name__ == '__main__': guarding the code that creates new processes (by the way, when you tag a question with multiprocessing , you should also tag the question with the platform being used), then any code at global scope will be executed by every process created to implement your pool. That means you should move if __name__ == '__main__': as follows:

if __name__ == '__main__':
    cap = cv2.VideoCapture('before2.mp4')
    _fourcc = cv2.VideoWriter_fourcc(*'MPEG')
    out = cv2.VideoWriter('after.mp4', _fourcc, 29.97, (1280,720))
    frames = []
    lst = []
    while cap.isOpened():
        ret, fullimg = cap.read()
        if not ret:
            break
        frames.append(fullimg)
        
        if len(frames) >= 8:
            with concurrent.futures.ProcessPoolExecutor() as executor:
                results = executor.map(function, frames)
                for i in results:
                    print(type(i))
                    out.write(i)
            frames.clear() 
    cap.release()
    out.release()
    cv2.destroyAllWindows() # destroy all opened windows

If you do not do this it seems to me that every sub-process in the pool will first attempt in parallel to create an empty video (the function worker function and out.write will never be called by these processes) and only then will the main process be able to invoke the function worker function using map . This doesn't quite explain why the main process doesn't succeed after all of these wasteful attempts. But...

You also have:

while cap.isOpened():

The documentations states that isOpened() returns True if the previous VideoCapture constructor succeeded. Then if this returns True once, why wouldn't it return True the next time it is tested and you end up looping indefinitely? Shouldn't the while be changed to an if ? And doesn't this suggest that isOpened() is perhaps returning False or else you would be looping indefinitely? Or what if len(frames) < 8 ? It seems then you would also end up with an empty output file.

My suggestion would be to make the above changes and try again.

Update

I took a closer look at the code more closely and it appears that it is looping reading the input ( before2.mp4 ) one frame at a time and when it has accumulated 8 frames or more it creates a pool and processes the frames it has accumulated and writing them out to the output ( after.mp4 ). But that means that if there are, for example, 8 more frames, it will create a brand new processing pool (very wasteful and expensive) and then write out the 8 additional processed frames. But if there were only 7 additional frames, they would never get processed and written out. I would suggest the following code (untested, of course):

def main():
    import os

    cap = cv2.VideoCapture('before2.mp4')
    if not cap.isOpened():
        return

    _fourcc = cv2.VideoWriter_fourcc(*'MPEG')
    out = cv2.VideoWriter('after.mp4', _fourcc, 29.97, (1280,720))

    FRAMES_AT_A_TIME = 8
    pool_size = min(FRAMES_AT_A_TIME, os.cpu_count())
    with concurrent.futures.ProcessPoolExecutor(max_workers=pool_size) as executor:
        more_frames = True

        while more_frames:
            frames = []
            for _ in range(FRAMES_AT_A_TIME):
                ret, fullimg = cap.read()
                if not ret:
                    more_frames = False
                    break
                frames.append(fullimg)

            if not frames:
                break # no frames        

            results = executor.map(function, frames)
            for i in results:
                print(type(i))
                out.write(i)

    cap.release()
    out.release()
    cv2.destroyAllWindows() # destroy all opened windows

if __name__ == '__main__':
    main()