Using tensorflow, how do I find the time taken for an epoch during fitting?

Question

I am following along this tutorial,https://www.tensorflow.org/tutorials/keras/basic_classification

When I fit the model using model.fit(train_images, train_labels, epochs=5, verbose =1) , the times are displayed in the python console. I want to get the wall time of each epoch by using time.clock() .

I am assuming that when more epochs are added the fitting time increases linearly, but I want to graph this to be sure.

Besides fitting with 1 epoch, then 2 epochs, then 3 epochs, etc, how can I work out the training time (fitting time) for an increasing number of epochs?

Answer 1

Using a custom callback you can plot the total time taken to fit certain epochs.

class timecallback(tf.keras.callbacks.Callback):
    def __init__(self):
        self.times = []
        # use this value as reference to calculate cummulative time taken
        self.timetaken = time.clock()
    def on_epoch_end(self,epoch,logs = {}):
        self.times.append((epoch,time.clock() - self.timetaken))
    def on_train_end(self,logs = {}):
        plt.xlabel('Epoch')
        plt.ylabel('Total time taken until an epoch in seconds')
        plt.plot(*zip(*self.times))
        plt.show()

And then pass this as a callback to the model.fit function like this

timetaken = timecallback()
model.fit(train_images, train_labels, epochs=5,callbacks = [timetaken])

This plots a graph at the end of training which shows the total time taken for the model to train upto a certain epoch from the start.

And if you want to plot the per epoch time. You can replace the on_train_end method with on_epoch_end.

def on_epoch_end(self,epoch,logs= {}):
    # same as the on_train_end function

Answer 2

Good practice to use callbacks. How do we record the durations in the history?

Say we're using history = model.fit( ..., callbacks=[my_training_callback])

What should I write into the definition of my_training_callback ? I'm tryin to do:

def my_training_callback(Callback):
    def __init__(self):
        mark = 0
        duration = 0
    def on_epoch_begin(self, epoch, logs=None):
        self.mark = time()
    def on_epoch_end(self, epoch, logs=None):
        self.duration = time() - self.mark

It works ok, but I'm having trouble adding the duration value to the history. Thanks

Answer 3

Using a custom callback definitely works but you must be careful of how you generate a timestamp. The recommended time.clock() works differently on Windows vs UNIX systems and may not generate the behavior that you want. Therefore, I recommend a tweak to the code others have recommended, using the built in tensorflow.timestamp() method ( documentation ). Note that this is a tensor object so if you'd like to plot the time as text, as I did, you'll need to extract the float value. I did so using .numpy() as this is an EagerTensor .

import tensorflow as tf
import matplotlib.pyplot as plt
from datetime import datetime

class timecallback(tf.keras.callbacks.Callback):
    def __init__(self):
        self.times = []
        self.epochs = []
        # use this value as reference to calculate cummulative time taken
        self.timetaken = tf.timestamp()
    def on_epoch_end(self,epoch,logs = {}):
        self.times.append(tf.timestamp() - self.timetaken)
        self.epochs.append(epoch)
    def on_train_end(self,logs = {}):
        plt.xlabel('Epoch')
        plt.ylabel('Total time taken until an epoch in seconds')
        plt.plot(self.epochs, self.times, 'ro')
        for i in range(len(self.epochs)):
          j = self.times[i].numpy()
          if i == 0:
            plt.text(i, j, str(round(j, 3)))
          else:
            j_prev = self.times[i-1].numpy()
            plt.text(i, j, str(round(j-j_prev, 3)))
        plt.savefig(datetime.now().strftime("%Y%m%d%H%M%S") + ".png")

Then, when calling model fit:

model.fit(train_images, train_labels, epochs=5,callbacks = [timecallback()])

Answer 4

I have managed to add the duration to the history.

The trick comes from the fact that the logs object passed to the methods of Callback is mutable, and is the same one that is passed to each callback... including the History object returned by model.fit .

So if you want the duration of each epoch in the history (as opposed to displayed on the screen or saved to a file as in the other answers), you have to add it to the logs object in your custom callback.

Example:

import datetime
import tensorflow as tf
class TimestampCallback(tf.keras.callbacks.Callback):
    def __init__(self, metric_name = "duration"):
        self.__epoch_start = None
        self.__metric_name = metric_name

    def on_epoch_begin(self, epoch, logs=None):
        self.__epoch_start = datetime.datetime.utcnow()

    def on_epoch_end(self, epoch, logs=None):
        logs[self.__metric_name] = datetime.datetime.utcnow() - self.__epoch_start

You don't have to use datetime obviously. And if you have a callback that consumes the duration, make sure to put it after this callback in the callback list. Because History is called after every callback, it will always receive the duration.