How do I terminate processes in an infinite loop in python?

Question

I have to write a code that converts video files from RGB to black and white using an equation that converts every frame to black and white. and I have to do that in parallel with multiprocessing and queue and with the help of Opencv. I did write the code but I have a problem with the termination of the processes in the infinite loop. How can I terminate the processes when I am finished with reading the frames, because the father is waiting for the children to finish and they never finish. this is my code..

#! /usr/bin/python
import numpy as np
import cv2
import multiprocessing as mp
import time


def read_frames(q1, q2):
    while True:
        NumAndFrame = q1.get()
        frame = NumAndFrame[1]
        if frame == 'Done':
            # Here is my problem,this is not working!!!
            processes.terminate()
            break
        j = NumAndFrame[0]
        R = frame[:, :, 0]
        G = frame[:, :, 1]
        B = frame[:, :, 2]
        y = (np.uint8)((0.299 * R) + (0.587 * G) + (0.114 * B))
        q2.put((j, y))



if __name__ == '__main__':
    start = time.time()
    q1 = mp.Queue()
    q2 = mp.Queue()
    processes = []
    for i in range(4):
        processes.append(mp.Process(target=read_frames, args=(q1, q2)))
    for p in processes:
        p.start()

    # feed the processes
    # read input file and send to the processes the frames:
    cap = cv2.VideoCapture('gou.avi')
    lines = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    cols = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    fourcc_ver = int(cap.get(cv2.CAP_PROP_FOURCC))
    out = cv2.VideoWriter('output.avi', fourcc_ver, fps, (cols, lines), False)

    j = 1
    while (cap.isOpened()):
        ret, frame = cap.read()
        # as long as new frames are there
        if ret == True:
            q1.put((j, frame))
            j += 1
        # if cv2.waitKey(1) & 0xFF == ord('q'):
        #   break
        else:
            break

q1.put((1, 'Done'))
for p in processes:
    p.join()
result = []
for p in processes:
    result.append(q2.get())
result.sort()
result = []
for r in result:
    result.append(r[1])
for i in result:
    out.write(i)
    print i
# Release everything if job is finished
print 'final finish'
cap.release()
out.release()
cv2.destroyAllWindows()

Answer 1

You might want to try to pair your question down to a smaller example, but if you're just interested in stopping the computation in the middle of a loop that is running indefinitely, you can spam Ctrl-C until it halts. Alternatively, you can just close the shell window.

Answer 2

Without having tested, for the same reason others gave in there comments: You should rather call "terminate" on each process within the main part, than call it in the child function:

 ...
for p in processes:
    p.terminate()
    p.join()

Answer 3

Consider using multiprocessing.Pool because it does most of the heavy lifting for you.

You need a "done" message for each child process. The child should send some sort of acknowledgement back to the parent and terminate. You also need some sort of error handling policy in the worker so that an exception doesn't just silently exit the worker.

You have other problems such as this code that confuses number of processes with number of messages processed.

for p in processes:
    result.append(q2.get())

Instead you should read all messages, counting the number of termination acknowledgements it gets on the way so that you know when to stop reading.

Your script is long and I'm not going to pretend that I've gotten it all right (please be friendly and post smaller examples in the future!) but here is a first go at cleaning it up.

#! /usr/bin/python
import numpy as np
import cv2
import multiprocessing as mp
import time


def read_frames(q1, q2):
    while True:
        try:
            NumAndFrame = q1.get()
            frame = NumAndFrame[1]
            if frame == 'Done':
                q2.put('Done')
                break
            j = NumAndFrame[0]
            R = frame[:, :, 0]
            G = frame[:, :, 1]
            B = frame[:, :, 2]
            y = (np.uint8)((0.299 * R) + (0.587 * G) + (0.114 * B))
            q2.put((j, y))
        except Exception, e:
            q2.put('Error: ' + str(e))


if __name__ == '__main__':
    start = time.time()
    q1 = mp.Queue()
    q2 = mp.Queue()
    processes = []
    for i in range(4):
        processes.append(mp.Process(target=read_frames, args=(q1, q2)))
    for p in processes:
        p.start()

    # feed the processes
    # read input file and send to the processes the frames:
    cap = cv2.VideoCapture('gou.avi')
    lines = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    cols = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    fourcc_ver = int(cap.get(cv2.CAP_PROP_FOURCC))
    out = cv2.VideoWriter('output.avi', fourcc_ver, fps, (cols, lines), False)

    j = 1
    while (cap.isOpened()):
        ret, frame = cap.read()
        # as long as new frames are there
        if ret == True:
            q1.put((j, frame))
            j += 1
        # if cv2.waitKey(1) & 0xFF == ord('q'):
        #   break
        else:
            break

for _ in len(processes):
    q1.put((1, 'Done'))
for p in processes:
    p.join()
result = []
done_count = 0
while done_count < len(processes):
    data = q2.get()
    if isinstance(data, basetring) and data == 'Done':
        done_count += 1
    else:
        result.append(data)

result.sort()

# What??? don't overwrite result here!
result = []
for r in result:
    result.append(r[1])
for i in result:
    out.write(i)
    print i
# Release everything if job is finished
print 'final finish'
cap.release()
out.release()
cv2.destroyAllWindows()

You end up holding the entire returned dataset in the parent so you may hit memory problems. And since (1) you have a large data payload being copied from parent to child and back, and (2) numpy releases the gil, you may find threads perform better than processes. You can check rather quickly by just substituting Thread for Process when you create the workers.