Python Image Processing Taking Too Long

Question

I would like to do image processing in Python. The problem is that I have a loop that is recording the image data from a camera into a numpy array, but in the loop I am trying to do a correlation of pixel data from the last image to the current image to determine whether or not I need to do further processing. This however is killing the execution speed of the loop which is showing lagged image output.

def gen(camera):
    image = np.zeros([480, 640, 3])
    last_image = np.zeros([480, 640, 3])

    frame_number = 0

    while True:
        frame = camera.get_frame()
        #new code

        if(frame_number % 10 == 0):
            #save the current frame
            image = frame
            #this line takes forever on 480x640x3 sized nd.arrray
            corr = signal.correlate(image, last_image)

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

        faces = faceCascade.detectMultiScale(
                    gray,
                    scaleFactor=1.1,
                    minNeighbors=5,
                    minSize=(30, 30),
                    flags=cv2.CASCADE_SCALE_IMAGE
                )

                # Draw a rectangle around the faces
        for (x, y, w, h) in faces:
                    cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)
        #cv2.imshow('Video', frame)
        ret, jpeg = cv2.imencode('.jpg', frame)
        #if cv2.waitKey(1) & 0xFF == ord('q'):
         #   break
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + jpeg.tobytes() + b'\r\n\r\n')
        if(frame_number % 10 == 0):
            #save current frame as last frame for next process 
            last_image = frame

        frame_number += 1

Is multi threading the way to solve this problem?

Answer 1

I am not sure if it is feasible, but you could make the operation faster if you correlate after converting to grayscale. This will make the correlation on a lot less data, and might actually be possible with decent FPS.

Another approach might be to correlate each channel independently, after a quick test, it might execute faster.

Alternatively, for a multithreaded approach you could correlate each of the RGB images on a different thread. How faster that would be, if faster at all, is left to find out with experiment. The link from jwpfox's comment might be a good start. However, last I was working with multiprocessing in Python, there was an issue of it copying the memory to each of the child processes. I do not know if this multiprocessing will do the same, but it is something to keep in mind for a future speed up.