How to work with multiple models in Tensorflow

Question

I have two models m1 and m2 trained separately. Now I want to keep m1 fixed and fine tune m1 based on the output of m2 . All the variables of m1 is under the variable scope "m1/" and the ones of m2 is under "m2/" . Here is basically what I did:

# build m1 and m2
with tf.device("/cpu:0"):
    m1.build_graph()
    m2.build_graph()
# indicate the variables of m1 and m2 
allvars = tf.global_variables()
m1_vars = [v for v in allvars if v.name.startswith('m1')]
m2_vars = [v for v in allvars if v.name.startswith('m2')]
# construct the saver 
m1_saver = tf.train.Saver(m1_vars)
m2_saver = tf.train.Saver(m2_vars)
# Load m2 variables 
m2_ckpt_state = tf.train.get_checkpoint_state(FLAGS.m2_log_root)
m2_sess = tf.Session()
m2_saver.restore(m2_sess, m2_ckpt_state.model_checkpoint_path)

# construct a train supervisor for m1
m1_sv = tf.train.Supervisor(is_chief=True, saver=m1_saver)
# construct a session for m1
m1_sess =  m1_sv.prepare_or_wait_for_session()
...

But now there is an error in the last line of code :

Traceback (most recent call last):
   File "run_summarization.py", line 407, in <module>
   tf.app.run()
   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
   File "run_summarization.py", line 401, in main run_fine_tune(model, ranker, batcher, vocab)
   File "run_summarization.py", line 232, in run_fine_tune sess_context_manager = sv.prepare_or_wait_for_session(config=config)
   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 719, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 280, in prepare_session
   self._local_init_op, msg))
   RuntimeError: Init operations did not make model ready.  Init op: init, 
   init fn: None, local_init_op: name: "group_deps"
   op: "NoOp"
   input: "^init_1"
   input: "^init_all_tables", error: Variables not initialized: m2/var1, m2/var2, m2/var3...

Could you please tell me why this error occurs and how can I fix it? Thanks in advance!

Answer 1

Use separate graph for separate models; here in this case supervisor is defined with m1_vars but it works with the default graph where m2_vars also reside, when trying to initialize m2_vars it causes the problem. Since m2_vars is initialized with another session.

function build_graph() should be defined as
    gi = tf.Graph()
    with gi.as_default():
         ... 
         rest of the code
    return gi
with tf.device("/cpu:0"):
    g1 = m1.build_graph()
    g2 = m2.build_graph()

...
m2_sess = tf.Session(graph=g2)
...
init_op = tf.variables_initializer(m2_vars)
m1_sv = tf.train.Supervisor(graph=g1, is_chief=True, init_op=init_op, saver=m1_saver)