How to use caffemodel with OpenCV on iOS?

Question

I am trying to use a .caffemodel alongside OpenCV on iOS devices. I found this github repository , but it can only be built with Xcode 6. I am working with Xcode 7, but I also downloaded Xcode 6 and still have no success on building it.

How can I use a caffemodel with OpenCV on iOS 9?

PS: The alternative would be this but it's written with swift & metal and I need to be able to use it with OpenCV.

Answer 1

You can use OpenCV DNN contrib module .

You need first to build OpenCV with contrib modules, you can find the steps here .

Then you can import and use the .caffemodel following this tutorial .

Here is an updated version of the tutorial, since it's not working as is:

#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
using namespace cv;
using namespace cv::dnn;
#include <fstream>
#include <iostream>
#include <cstdlib>
using namespace std;
/* Find best class for the blob (i. e. class with maximal probability) */
void getMaxClass(dnn::Blob &probBlob, int *classId, double *classProb)
{
    Mat probMat = probBlob.matRefConst().reshape(1, 1); //reshape the blob to 1x1000 matrix
    Point classNumber;
    minMaxLoc(probMat, NULL, classProb, NULL, &classNumber);
    *classId = classNumber.x;
}
std::vector<String> readClassNames(const char *filename = "synset_words.txt")
{
    std::vector<String> classNames;
    std::ifstream fp(filename);
    if (!fp.is_open())
    {
        std::cerr << "File with classes labels not found: " << filename << std::endl;
        exit(-1);
    }
    std::string name;
    while (!fp.eof())
    {
        std::getline(fp, name);
        if (name.length())
            classNames.push_back( name.substr(name.find(' ')+1) );
    }
    fp.close();
    return classNames;
}
int main(int argc, char **argv)
{
    cv::dnn::initModule();        

    String modelTxt = "bvlc_googlenet.prototxt";
    String modelBin = "bvlc_googlenet.caffemodel";
    String imageFile = (argc > 1) ? argv[1] : "space_shuttle.jpg";
    Ptr<dnn::Importer> importer;
    try                                     //Try to import Caffe GoogleNet model
    {
        importer = dnn::createCaffeImporter(modelTxt, modelBin);
    }
    catch (const cv::Exception &err)        //Importer can throw errors, we will catch them
    {
        std::cerr << err.msg << std::endl;
    }
    if (!importer)
    {
        std::cerr << "Can't load network by using the following files: " << std::endl;
        std::cerr << "prototxt:   " << modelTxt << std::endl;
        std::cerr << "caffemodel: " << modelBin << std::endl;
        std::cerr << "bvlc_googlenet.caffemodel can be downloaded here:" << std::endl;
        std::cerr << "http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel" << std::endl;
        exit(-1);
    }
    dnn::Net net;
    importer->populateNet(net);
    importer.release();                     //We don't need importer anymore
    Mat img = imread(imageFile);
    if (img.empty())
    {
        std::cerr << "Can't read image from the file: " << imageFile << std::endl;
        exit(-1);
    }
    resize(img, img, Size(224, 224));                   //GoogLeNet accepts only 224x224 RGB-images
    dnn::Blob inputBlob = dnn::Blob(img);   //Convert Mat to dnn::Blob batch of images
    net.setBlob(".data", inputBlob);        //set the network input
    net.forward();                          //compute output
    dnn::Blob prob = net.getBlob("prob");   //gather output of "prob" layer
    int classId;
    double classProb;
    getMaxClass(prob, &classId, &classProb);//find the best class
    std::vector<String> classNames = readClassNames();
    std::cout << "Best class: #" << classId << " '" << classNames.at(classId) << "'" << std::endl;
    std::cout << "Probability: " << classProb * 100 << "%" << std::endl;
    return 0;
} //main

Answer 2

I will post another answer because with recent versions there are some differences.

First of all now dnn is already inside the standard OpenCV library so you do not have to build it from contrib_modules .

The function to load the network is readNetFromCaffe .

For example the following code load the NN:

  std::string modelName = "path/to/mymodel.caffemodel";

  std::string protoName = "path/to/deploy.prototxt";
  cv::dnn::Net net;
  try
  {
    net = cv::dnn::readNetFromCaffe(protoName, modelName);
  }
  catch (cv::Exception& e)
  {
    std::cerr << "Exception: " << e.what() << std::endl;
    if (net.empty())
    {
      std::cerr << "Can't load network by using the following files: " << std::endl;
      std::cerr << "prototxt:   " << protoName << std::endl;
      std::cerr << "caffemodel: " << modelName << std::endl;
      std::cerr << "bvlc_googlenet.caffemodel can be downloaded here:" << std::endl;
      std::cerr << "http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel" << std::endl;
      exit(-1);
    }
  }

Then you can run the NN:

  cv::Mat res_mat;
  float res;
  cv::Mat inputBlob = cv::dnn::blobFromImage(roi, 1.0f, cv::Size(227, 227),
                                  cv::Scalar(0, 0, 0), false);
  net.setInput(inputBlob);
  //During the forward pass output of each network layer is computed,
  //but in this example we need output from "prob" layer only.
  res_mat = net.forward("score");
  std::cout<<res_mat<<std::endl;
  res_mat = res_mat.reshape(1, 1); //reshape the blob to 1x2 matrix
  return res_mat.at<float>(0); 

The function cv::dnn::blobFromImage resize the image to the input network size specified by the third argument (in my case cv::Size(227, 227) ). The argument cv::Scalar(0, 0, 0) is to subtract the means from the three BGR channels.

score is the name of the output layer in the NN I used. You can see this information in the prototxt file.