OpenCV & Python - Real time image (frame) processing

Question

We're doing a project in school where we need to do basic image processing. Our goal is to use every video frame for the Raspberry Pi and do real time image processing.

We've tried to include raspistill in our python-program but so far nothing has worked. The goal of our project is to design a RC-car which follows a blue/red/whatever coloured line with help from image processing.

We thought it would be a good idea to make a python-program which does all image processing necessary, but we currently struggle with the idea of bringing recorded images into the python program. Is there a way to do this with picamera or should we try a different way?

For anyone curious, this is how our program currently looks

while True:
    #camera = picamera.PiCamera()
    #camera.capture('image1.jpg')
    img = cv2.imread('image1.jpg')
    width = img.shape[1]
    height = img.shape[0]
    height=height-1
    for x in range (0,width):
            if x>=0 and x<(width//2):
                    blue  = img.item(height,x,0)
                    green = img.item(height,x,1)
                    red   = img.item(height,x,2)
                    if red>green and red>blue:

Answer 1

OpenCV already contains functions to process live camera data.

This OpenCV documentation provides a simple example:

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Of course, you do not want to show the image but all your processing can be done there.

Remember to sleep a few hundred milliseconds so the pi does not overheat that much.

Edit:

"how exactly would I go about it though. I used "img = cv2.imread('image1.jpg')" all the time. What do I need to use instead to get the "img" variable right here? What do I use? And what is ret, for? :)"

ret indicates whether the read was successful. Exit program if not.

The read frame is nothing other than your img = cv2.imread('image1.jpg') so your detection code should work exactly the same.

The only difference is that your image does not need to be saved and reopened. Also for debugging purposes you can save the recorded image, like:

import cv2, time

cap = cv2.VideoCapture(0)

ret, frame = cap.read()
if ret:
    cv2.imwrite(time.strftime("%Y%m%d-%H%M%S"), frame)

cap.release()

Answer 2

You can use picamera to acquire images .

To make it "real time", you can acquire data each X milliseconds. You need to set X depending on the power of your hardware (and the complexity of the openCV algorithm).

Here's an example (from http://picamera.readthedocs.io/en/release-1.10/api_camera.html#picamera.camera.PiCamera.capture_continuous ) how to acquire 60 images per second using picamera:

import time
import picamera
with picamera.PiCamera() as camera:
    camera.start_preview()
    try:
        for i, filename in enumerate(camera.capture_continuous('image{counter:02d}.jpg')):
            print(filename)
            time.sleep(1)
            if i == 59:
                break
    finally:
        camera.stop_preview()

Answer 3

Sometimes you want to control the "latency" or delay between grabbing an image, processing it and then doing something with the result. For example, if it's part of a height measurement system and you need to deliver a height result at exactly 10Hz , in synchronisation with other sensors or actuators, you might want to use the time functions. Most commercial and real-world systems produce data at regular intervals like this - although it might not be important for a hobby project of course !