TensorFlow：在MonitoredSession中恢复模型

Question

I have a model that contains multiple variables including a global step. I've been able to successfully use a MonitoredSession to save checkpoints and summaries every 100 steps. I was expecting the MonitoredSession to automatically restore all my variables when the session is run in multiple passes (based on this documentation), however this does not happen. If I take a look at the global step after running the training session again, I find that it starts back from zero. This is a simplified version of my code without the actual model. Let me know if more code is needed to solve this problem

train_graph = tf.Graph()
with train_graph.as_default():
  # I create some datasets using the Dataset API
  # ...

  global_step = tf.train.create_global_step()

  # Create all the other variables and the model here
  # ...

  saver_hook = tf.train.CheckpointSaverHook(
      checkpoint_dir='checkpoint/',
      save_secs=None,
      save_steps=100,
      saver=tf.train.Saver(),
      checkpoint_basename='model.ckpt',
      scaffold=None)
  summary_hook = tf.train.SummarySaverHook(
      save_steps=100,
      save_secs=None,
      output_dir='summaries/',
      summary_writer=None,
      scaffold=None,
      summary_op=train_step_summary)
  num_steps_hook = tf.train.StopAtStepHook(num_steps=500) # Just for testing


  with tf.train.MonitoredSession(
      hooks=[saver_hook, summary_hook, num_steps_hook]) as sess:
    while not sess.should_stop():
      step = sess.run(global_step)
      if (step % 100 == 0):
        print(step)
      sess.run(optimizer)

When I run this code the first time, I get this output

0
100
200
300
400

The checkpoint folder at this point has checkpoints for every hundredth step up to 500. If I run the program again I would expect to see the counter start at 500 and the increase up to 900, but instead I just get the same thing and all of my checkpoints get overwritten. Any ideas?

Answer 1

Alright, I figured it out. It was actually really simple. First, it's easier to use a MonitoredTraningSession() instead of a MonitoredSession(). This wrapper session takes as an argument 'checkpoint_dir'. I thought that the saver_hook would take care of restoring, but that's not the case. In order to fix my problem I just had to change the line where I define the session like so:

with tf.train.MonitoredTrainingSession(hooks=[saver_hook, summary_hook], checkpoint_dir='checkpoint'):

It can also be done with the MonitoredSession directly, but you need to set up a session_creator instead.