Tensorflow: Is there a way to store the training loss in tf.Estimator

Question

I am using a tensorflow estimator object to train a model from the official tensorflow layers documentation ( https://www.tensorflow.org/tutorials/layers ). I can see that the training loss is displayed on the console during training. Is there a way to store these training loss values?

Thanks!

Answer 1

The displaying is done via logging.info . tf.estimator creates a LoggingTensorHook for the training loss to do this, see here .

I suppose you could reroute the logging output to some file, but this would still not give you the raw values.

Two ways I could think of:

1. Write your own hook to store the values; this would probably look extremely similar to LoggingTensorHook , you would just need to write the numbers to a file instead of printing them.
2. By default tf.estimator also creates summary data in Tensorboard for the training loss; you could open the "Scalar" tab in Tensorboard where you should see the loss curve. Tick "Show data download links" in the top left corner. This will give you an option to download each graph's data in either CSV or JSON format. By default, both the logging and summary hooks are set up such that they both log values every 100 steps. So the graph should have the same information you saw in the console. If you're unfamiliar with Tensorboard, there are tutorials on the Tensorflow website as well; the basic usage should be quite simple!

Answer 2

You can use TensorBoard event file in model_dir after training your estimator by estimator.train()

model = tf.estimator.Estimator(..., model_dir= 'tmp') 
# model data will be save in tmp directory after training

image

The event file has the name events.out.tfevents.15121254...., this file save the log of training process (there is an other event file in eval folder that save evaluate log). You can get training loss by:

for e in tf.train.summary_iterator(path_to_events_file):
for v in e.summary.value:
    if v.tag == 'loss':
        print(v.simple_value)

In addition, you can save other values during training by add tf.summary inside your model_fn:

tf.summary.scalar('accuracy', accuracy)

reference: https://www.tensorflow.org/api_docs/python/tf/train/summary_iterator