Weights saved in (Python) tensorflow not loaded in C++ tf

Question

I've followed along with the linked posts and finally managed to load my trained tensorflow model's graph+weights into C++ without it throwing errors at me, but it seems to not properly load the weights. It's possible I'm missing a step, most likely in the inference section.

I've included a fully functional minimal working example below. In order to run it you'll (probably) need Python + 1.4.0 < Tensorflow <= 1.7.0, Tensorflow for C++, and keras. Any insights into the issue I'm having would be very happily received, I've been in an uphill battle with tf for 2+weeks now.

Python code which will "train" a model and save it + weights as shown here :

from __future__ import absolute_import, division, print_function
import os
import tensorflow as tf
import numpy as np
import random
import keras
from keras import backend as K

# Set it manually so C++ interface can use mem growth
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
K.set_session(sess)

model = tf.keras.Sequential([
    tf.keras.layers.Convolution2D(64, (5, 5), input_shape=(7, 19, 19), data_format='channels_first', name='Input'),
    tf.keras.layers.ZeroPadding2D(padding =(4, 4), data_format='channels_first', name='Pad0'),
    tf.keras.layers.BatchNormalization(axis=1, momentum=0.99, name='Norm0'), 
    tf.keras.layers.Dropout(0.25, name='Drop0'),
    tf.keras.layers.Flatten(name='Flatten0'),
    tf.keras.layers.Dense(361, activation='softmax', name='Output'),
])

X = np.ones((25, 7, 19, 19))
Y = np.zeros((25, 361))

optimizer = tf.train.AdamOptimizer(learning_rate=0.0018)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

x = np.zeros((1, 7, 19, 19))
print(model.predict(x))

model.fit(X, Y, 1, 5, 2)
print(model.predict(x)) # Just to verify output is different after training

K.set_learning_phase(0)
sess.run(tf.global_variables_initializer())

saver = tf.train.Saver(tf.global_variables())
saver.save(sess, save_path='./models/myModel')
// Just used to look at graph structure
tf.train.write_graph(sess.graph, '.', './models/graph.pb', as_text=True)

And C++ code, currently using tf 1.5.0:

Code to load the model as shown here

#define _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS
#define COMPILER_MSVC
#define NOMINMAX
#include <iomanip>
#include <tensorflow\core\public\session.h>
#include <tensorflow\core\protobuf\meta_graph.pb.h>
#include <tensorflow\core\framework\tensor.h>
#include <tensorflow\cc\ops\standard_ops.h>

namespace tf = tensorflow;
tf::Status status;
tf::Session* session;
tf::SessionOptions options;
tf::MetaGraphDef graphDef;
std::string pathToGraph = "models/myModel.meta";
std::string pathToCheckpoint = "models/myModel";

int main()
{
    options.config.mutable_gpu_options()->set_allow_growth(true);
    options.config.mutable_gpu_options()->set_per_process_gpu_memory_fraction(1.0);

    session = tf::NewSession(options);
    if (!session)
         throw std::runtime_error("Could no create Tensorflow Session!");

    // Read in the protobuf
    status = tf::ReadBinaryProto(tf::Env::Default(), pathToGraph, &graphDef);
    if (!status.ok())
        throw std::runtime_error("Error reading graph: " + status.ToString());

    status = session->Create(graphDef.graph_def());
    if (!status.ok())
        throw std::runtime_error("Error creating graph: " + status.ToString());

    // Read the weights
    tf::Tensor checkpointPathTensor(tf::DT_STRING, tf::TensorShape());
    checkpointPathTensor.scalar<std::string>()() = pathToCheckpoint;

    const auto fileTensorName = graphDef.saver_def().filename_tensor_name();
    const auto restoreOpName = graphDef.saver_def().restore_op_name();

    status = session->Run(
        { { fileTensorName, checkpointPathTensor }, },
        {},
        { restoreOpName },
        nullptr
    );

    if (!status.ok())
        throw std::runtime_error("Error loading checkpoint from " + pathToCheckpoint + ": " + status.ToString());

    ...

Followed by the code to run inference with the loaded/trained model

    float inData[2527] = { 0.f };

    static const std::string inputName = "Input_input";
    static const std::string outputName = "Output/Softmax";

    static const auto shape = tf::TensorShape({ 1, 7, 19, 19});
    tf::Tensor input(tf::DT_FLOAT, shape);
    std::copy_n(inData, 2527, input.flat<float>().data());

    std::vector<tf::Tensor> outputs;
    status = session->Run({ { inputName, input } }, { outputName }, {}, &outputs);

    tf::TTypes<float>::Flat flatOut = outputs[0].flat<float>();

    for (int i = 0; i < 361; ++i)
    {
        if (i % 19 == 0)
            std::cout << '\n';
        std::cout << std::setw(8) << std::fixed << std::setprecision(8) << flatOut(i) << ", ";
    }

    std::cout << '\n';

}

For me, when I run this everything works as it should in the Python section but silently fails to load a trained graph (or does something that looks similar) in the C++ section. When running the C++ section the model outputs exactly the same output as the non-trained Python model does. Is there some step I'm missing here?

Sample Python output before model training:

[[0.00277008 0.00277008 0.00277008 0.00277008 0.00277008 0.00277008 
0.00277008 0.00277008 0.00277008 0.00277008 0.00277008  ...etc ]]

Sample output after training in Python:

.00387822 0.00228055 0.0018196  0.0014322  0.00266262 0.00234695 
0.0026182  0.00322318 0.00252047 0.00353322 0.00342526 ...etc ]]

Sample output after loading model and running in C++ with identical inputs:

0.00277008, 0.00277008, 0.00277008, 0.00277008, ... etc

Ideally I'd like to be seeing C++ output that is identical to the Python output after training!

Answer 1

You can modify main.cc from tf_label_image_example project.

However, I set the tensorflow_BUILD_CC_EXAMPLE=ON when I built the tensorflow using CMake.

You can reference this http://www.stefanseibert.com/2017/10/tensorflow-as-dll-into-your-windows-c-project-with-gpu-support-and-cmake-v1-3/ .

In addition, I built a single Visual Studio Project(VS 2015 x64 Release) which is similar with tf_label_image_example_project.

Actually, I opened tf_label_image_example_project using Visual Studio, and copied many configurations into my project. Most importantly, the Object Libraries in *.vcxproj and *.vcxproj.filters.