OpenCV 在 Google Colaboratory 中不起作用

Question

I was practicing OpenCV on google colaboratory becasuse I don't know how to use OpenCV on GPU, when I run OpenCV on my hardware, It takes a lot of CPU, so I went to Google colaboratory. The link to my notebook is here .

If you don't want to watch it, then here is the code:

import cv2

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
cap = cv2.VideoCapture(0)

while True:
    _, img = cap.read()
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.1, 4)
    for (x, y, w, h) in faces:
        cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)

    cv2.imshow('img', img)

    k = cv2.waitKey(30) & 0xff
    if k==27:
        break
    
cap.release()

The same code worked fine on my PC, but not on Google Colaboratory. The error is:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-5-0d9472926d8c> in <module>()
      6 while True:
      7         _, img = cap.read()
----> 8         gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
      9         faces = face_cascade.detectMultiScale(gray, 1.1, 4)
     10         for (x, y, w, h) in faces:

error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

PS~I have the haarcascade file inside the same directory of my notebook in Google Colaboratory

How to deal with it? If not then is there any " concrete " solution to run OpenCV on my CUDA enabled GPU instead of CPU? Thanks in advance!

Answer 1

_src.empty() means that it had problem to get frame from camera and img is None and when it tries cvtColor(None, ...) then it gives _src.empty() .

You should check if img is not None: because cv2 doesn't raise error when it can't get frame from camera or read image from file. And sometimes camera needs time to "warm up" and it can gives few empty frames ( None ).

---

VideoCapture(0) reads frame from camera directly connected to computer which runs this code - and when you run code on server Google Colaboratory then it means camera connected directly to server Google Colaboratory (not your local camera) but this server doesn't have camera so VideoCapture(0) can't work on Google Colaboratory .

cv2 can't get image from your local camera when it runs on server. Your web browser may have access to your camera but it needs JavaScript to get frame and send to server - but server needs code to get this frame

---

I checked in Google if Google Colaboratory can access local webcam and it seems they created script for this - Camera Capture - in first cell is function take_photo() which uses JavaScript to access your camera and display in browser, and in second cell this function is used to display image from local camera and to take screenshot.

You should use this function instead of VideoCapture(0) to work on server with your local camera.

---

BTW: Belove take_photo() there is also information about cv2.im_show() because it also works only with monitor directly connected to computer which runs this code (and this computer has to run GUI like Windows on Windows , X11 on Linux) - and when you run it on server then it want to display on monitor directly connected to server - but server usually works without monitor (and without GUI)

Google Colaboratory has special replacement which displays in web browser

 from google.colab.patches import cv2_imshow

---

BTW: If you will have problem with loading haarcascades .xml then you may need folder to filename. cv2 has special variable for this cv2.data.haarcascades

path = os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml')

cv2.CascadeClassifier( path )

You can also see what is in this folder

import os

filenames = os.listdir(cv2.data.haarcascades)
filenames = sorted(filenames)
print('\n'.join(filenames))

---

EDIT:

I created code which can get from local webcam frame by frame without using button and without saving in file. Problem is that it is slow - because it still have to send frame from local web browser to google colab server and later back to local web browser

Python code with JavaScript functions

#
# based on: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=2viqYx97hPMi
#

from IPython.display import display, Javascript
from google.colab.output import eval_js
from base64 import b64decode, b64encode
import numpy as np

def init_camera():
  """Create objects and functions in HTML/JavaScript to access local web camera"""

  js = Javascript('''

    // global variables to use in both functions
    var div = null;
    var video = null;   // <video> to display stream from local webcam
    var stream = null;  // stream from local webcam
    var canvas = null;  // <canvas> for single frame from <video> and convert frame to JPG
    var img = null;     // <img> to display JPG after processing with `cv2`

    async function initCamera() {
      // place for video (and eventually buttons)
      div = document.createElement('div');
      document.body.appendChild(div);

      // <video> to display video
      video = document.createElement('video');
      video.style.display = 'block';
      div.appendChild(video);

      // get webcam stream and assing to <video>
      stream = await navigator.mediaDevices.getUserMedia({video: true});
      video.srcObject = stream;

      // start playing stream from webcam in <video>
      await video.play();

      // Resize the output to fit the video element.
      google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);

      // <canvas> for frame from <video>
      canvas = document.createElement('canvas');
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      //div.appendChild(input_canvas); // there is no need to display to get image (but you can display it for test)

      // <img> for image after processing with `cv2`
      img = document.createElement('img');
      img.width = video.videoWidth;
      img.height = video.videoHeight;
      div.appendChild(img);
    }

    async function takeImage(quality) {
      // draw frame from <video> on <canvas>
      canvas.getContext('2d').drawImage(video, 0, 0);

      // stop webcam stream
      //stream.getVideoTracks()[0].stop();

      // get data from <canvas> as JPG image decoded base64 and with header "data:image/jpg;base64,"
      return canvas.toDataURL('image/jpeg', quality);
      //return canvas.toDataURL('image/png', quality);
    }

    async function showImage(image) {
      // it needs string "data:image/jpg;base64,JPG-DATA-ENCODED-BASE64"
      // it will replace previous image in `<img src="">`
      img.src = image;
      // TODO: create <img> if doesn't exists, 
      // TODO: use `id` to use different `<img>` for different image - like `name` in `cv2.imshow(name, image)`
    }

  ''')

  display(js)
  eval_js('initCamera()')

def take_frame(quality=0.8):
  """Get frame from web camera"""

  data = eval_js('takeImage({})'.format(quality))  # run JavaScript code to get image (JPG as string base64) from <canvas>

  header, data = data.split(',')  # split header ("data:image/jpg;base64,") and base64 data (JPG)
  data = b64decode(data)  # decode base64
  data = np.frombuffer(data, dtype=np.uint8)  # create numpy array with JPG data

  img = cv2.imdecode(data, cv2.IMREAD_UNCHANGED)  # uncompress JPG data to array of pixels

  return img

def show_frame(img, quality=0.8):
  """Put frame as <img src="data:image/jpg;base64,...."> """

  ret, data = cv2.imencode('.jpg', img)  # compress array of pixels to JPG data

  data = b64encode(data)  # encode base64
  data = data.decode()  # convert bytes to string
  data = 'data:image/jpg;base64,' + data  # join header ("data:image/jpg;base64,") and base64 data (JPG)

  eval_js('showImage("{}")'.format(data))  # run JavaScript code to put image (JPG as string base64) in <img>
                                           # argument in `showImage` needs `" "` 

And code which uses it in loop

# 
# based on: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb#scrollTo=zo9YYDL4SYZr
#

#from google.colab.patches import cv2_imshow  # I don't use it but own function `show_frame()`

import cv2
import os

face_cascade = cv2.CascadeClassifier(os.path.join(cv2.data.haarcascades, 'haarcascade_frontalface_default.xml'))

# init JavaScript code
init_camera()

while True:
    try:
        img = take_frame()

        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        #cv2_imshow(gray)  # it creates new image for every frame (it doesn't replace previous image) so it is useless
        #show_frame(gray)  # it replace previous image

        faces = face_cascade.detectMultiScale(gray, 1.1, 4)

        for (x, y, w, h) in faces:
                cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)
        
        #cv2_imshow(img)  # it creates new image for every frame (it doesn't replace previous image) so it is useless
        show_frame(img)  # it replace previous image
        
    except Exception as err:
        print('Exception:', err)

I don't use from google.colab.patches import cv2_imshow because it always add new image on page instead of replacing existing image.

---

The same code as Notebook on Google Colab:

https://colab.research.google.com/drive/1j7HTapCLx7BQUBp3USiQPZkA0zBKgLM0?usp=sharing

Answer 2

The possible problem in the code is, you need to give full-path as the directory when using Haar-like features.

face_cascade = cv2.CascadeClassifier('/User/path/to/opencv/data/haarcascades/haarcascade_frontalface_default.xml')

The colab issue with opencv has been known for quite some time, also the same question asked here

As stated here , you can use the cv2_imshow to display the image, but you want to process Camera frames.

from google.colab.patches import cv2_imshow
img = cv2.imread('logo.png', cv2.IMREAD_UNCHANGED)
cv2_imshow(img)

One possible answer:

Insert Camera Capture snippet, the method take_photo but you need to modify the method.

face_cascade = cv2.CascadeClassifier('/opencv/data/haarcascades/haarcascade_frontalface_default.xml')

try:
    filename = take_photo()
    img = Image(filename)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.1, 4)
    for (x, y, w, h) in faces:
            cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)
    cv2_imshow("img", img)
      
except Exception as err:
    print(str(err))

The above code requires editing since there is no direct way to use VideoCapture you have to modify take_photo