tensorflow在图执行期间添加摘要

Question

I want to output the accuracy every fixed amount of steps in the cifar10 example given by the tensorflow tutorial, I tried using the tf.summary.scalar(..) in the hook which produces the error: Graph is finalized. However, I think I can only access the number of steps in the hook (I'm evaluating the accuracy use cifar10_eval.py, also an example code given by the tensorflow tutorial). I also tried to write the global_step into the checkpoint but unfortunately the MointeredTrainingSession only supports the time interval( save_checkpoint_secs ) instead of step interval. Any suggestions?

cifar10_train.py

def train():
  """Train CIFAR-10 for a number of steps."""
  with tf.Graph().as_default():
    global_step = tf.contrib.framework.get_or_create_global_step()

    # Build a Graph that trains the model with one batch of examples and
    # updates the model parameters.
    train_op = cifar10.train(loss, global_step)

    class _LoggerHook(tf.train.SessionRunHook):
      """Logs loss and runtime."""

      def begin(self):
        self._step = -1
        self._start_time = time.time()

      def before_run(self, run_context):
        self._step += 1
        return tf.train.SessionRunArgs(loss)  # Asks for loss value.

      def after_run(self, run_context, run_values):
        <output some information>

    with tf.train.MonitoredTrainingSession(
        checkpoint_dir=FLAGS.train_dir,
        hooks=[tf.train.StopAtStepHook(last_step=FLAGS.max_steps),
               tf.train.NanTensorHook(loss),
               _LoggerHook()],
        config=tf.ConfigProto(
            log_device_placement=FLAGS.log_device_placement)) as mon_sess:
      while not mon_sess.should_stop():
        mon_sess.run(train_op)

Answer 1

First of all, it should be noted that cifar10 tutorial provided by Tensorflow runs training and evaluating on two separate sessions. When training session saves a checkpoint, evaluating session will retrieve this chekpoint. Then load parameters and perform evaluating. Code you paste here is just for training session.

My advise is, you should make it clear which summary are you going to write. Because training and evaluating are two different sessions. There are two summary writers. And usually, they will provide different paths for different summary writers.

Based on your need, here are some hints for your project.

• You should not write anything to checkpoint, which is bulk of model parameters.
• Please use summary writer or standard i/o for recording accuracy.
• You got an error when you tried to use summary writer because all elements of summary including scalar should be added before session is launched.

I guess Tensorflow regards summary as a part of default graph. So you may want to configure your summary writer before you run a session.

Answer 2

I got this error when I worked with CycleGAN before. I solved this issue with these 2 lines, please add this before initializing the tf.

import tensorflow as tf

tf.reset_default_graph()
tf.Graph().as_default()

I hope this will help you.