遮罩RCNN OpenVino-C ++ API

Question

I would like to implement a custom image classifier using MaskRCNN.

In order to increase the speed of the network, i would like to optimise the inference.

I already used OpenCV DNN library, but i would like to do a step forward with OpenVINO.

I used successfully OpenVINO Model optimiser (python), to build the .xml and .bin file representing my network.

I successfully builded OpenVINO Sample directory with Visual Studio 2017 and run MaskRCNNDemo project.

mask_rcnn_demo.exe -m .\Release\frozen_inference_graph.xml -i .\Release\input.jpg

InferenceEngine:
        API version ............ 1.4
        Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ]     .\Release\input.jpg
[ INFO ] Loading plugin

        API version ............ 1.5
        Build .................. win_20181005
        Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ WARNING ] Image is resized from (4288, 2848) to (800, 800)
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Start inference (1 iterations)

Average running time of one iteration: 2593.81 ms

[ INFO ] Processing output blobs
[ INFO ] Detected class 16 with probability 0.986519: [2043.3, 1104.9], [2412.87, 1436.52]
[ INFO ] Image out.png created!
[ INFO ] Execution successful

Then i tried to reproduce this project in a separate project... First i had to watch dependancies...

<MaskRCNNDemo>
     //References
     <format_reader/>    => Open CV Images, resize it and get uchar data
     <ie_cpu_extension/> => CPU extension for un-managed layers (?)

     //Linker
     format_reader.lib         => Format Reader Lib (VINO Samples Compiled)
     cpu_extension.lib         => CPU extension Lib (VINO Samples Compiled)
     inference_engined.lib     => Inference Engine lib (VINO)
     opencv_world401d.lib      => OpenCV Lib
     libiomp5md.lib            => Dependancy
     ... (other libs)

With it i've build a new project, with my own classes and way to open images (multiframe tiff). This work without problem then i will not describe (i use with a CV DNN inference engine without problem).

I wanted to build the same project than MaskRCNNDemo : CustomIA

<CustomIA>
     //References
     None => I use my own libtiff way to open image and i resize with OpenCV
     None => I will just add include to cpu_extension source code.

     //Linker
     opencv_world345d.lib   => OpenCV 3.4.5 library
     tiffd.lib              => Libtiff Library
     cpu_extension.lib      => CPU extension compiled with sample
     inference_engined.lib  => Inference engine lib.

I added the following dll to the project target dir :

cpu_extension.dll
inference_engined.dll
libiomp5md.dll
mkl_tiny_omp.dll
MKLDNNPlugind.dll
opencv_world345d.dll
tiffd.dll
tiffxxd.dll

I successfully compiled and execute but i faced two issues :

OLD CODE:

 slog::info << "Loading plugin" << slog::endl;
    InferencePlugin plugin = PluginDispatcher({ FLAGS_pp, "../../../lib/intel64" , "" }).getPluginByDevice(FLAGS_d);

    /** Loading default extensions **/
    if (FLAGS_d.find("CPU") != std::string::npos) {
        /**
         * cpu_extensions library is compiled from "extension" folder containing
         * custom MKLDNNPlugin layer implementations. These layers are not supported
         * by mkldnn, but they can be useful for inferring custom topologies.
        **/
        plugin.AddExtension(std::make_shared<Extensions::Cpu::CpuExtensions>());
    }
    /** Printing plugin version **/
    printPluginVersion(plugin, std::cout);

OUTPUT :

[ INFO ] Loading plugin
    API version ............ 1.5
    Build .................. win_20181005
    Description ....... MKLDNNPlugin

NEW CODE:

    VINOEngine::VINOEngine()
{
    // Loading Plugin
    std::cout << std::endl;
    std::cout << "[INFO] - Loading VINO Plugin..." << std::endl;
    this->plugin= PluginDispatcher({ "", "../../../lib/intel64" , "" }).getPluginByDevice("CPU");
    this->plugin.AddExtension(std::make_shared<Extensions::Cpu::CpuExtensions>());
    printPluginVersion(this->plugin, std::cout);

OUTPUT :

[INFO] - Loading VINO Plugin...
000001A242280A18  // Like memory adress ???

Second Issue :

When i try to extract my ROI and masks from New Code, if i have a "match", i always have :

• score =1.0
• x1=x2=0.0
• y1=y2=1.0

But the mask looks well extracted...

New Code :

        float score = box_info[2];
        if (score > this->Conf_Threshold)
        {
            // On reconstruit les coordonnées de la box..
            float x1 = std::min(std::max(0.0f, box_info[3] * Image.cols), static_cast<float>(Image.cols));
            float y1 = std::min(std::max(0.0f, box_info[4] * Image.rows), static_cast<float>(Image.rows));
            float x2 = std::min(std::max(0.0f, box_info[5] * Image.cols), static_cast<float>(Image.cols));
            float y2 = std::min(std::max(0.0f, box_info[6] * Image.rows), static_cast<float>(Image.rows));
            int box_width = std::min(static_cast<int>(std::max(0.0f, x2 - x1)), Image.cols);
            int box_height = std::min(static_cast<int>(std::max(0.0f, y2 - y1)), Image.rows);

Image is resized from (4288, 2848) to (800, 800)
Detected class 62 with probability 1: [4288, 0], [4288, 0]

Then it is impossible for me to place the mask in the image and resize it while i don't have correct bbox coordinate...

Do anybody have an idea about what i make badly ?

How to create and link correctly an OpenVINO project using cpu_extension ?

Thanks !

Answer 1

First issue with version: look above printPluginVersion function, you will see overloaded std::ostream operators for InferenceEngine and plugin version info.

Second: You can try to debug your model by comparing output after very first convolution and output layer for original framework and OV. Make sure it's equal element by element.

In OV you can use network.addOutput("layer_name") to add any layer to output. Then read output by using: const Blob::Ptr debug_blob = infer_request.GetBlob("layer_name").

Most of the time with issues like this i finding missing of input pre-processing (mean, normalization, etc.)

cpu_extensions is a dynamic library, but you still can change cmake script to make it static and link it with your application. After that you would need to use your application path with call to IExtensionPtr extension_ptr = make_so_pointer(argv[0])