tensorflow calculating loss with GPU vs. CPU

Question

I'm a bit confused with the difference I'm getting in loss calculations when calculating loss with a GPU vs. CPU.

Model is a 6 layer CNN.

Loaded the model from checkpoints and ran it with the same data. Calculated the loss with a CPU and then a GPU.

CPU loss: 0.4687191 GPU loss: 0.46873742

Could someone explain to me why these losses were calculated differently?

#WITH CPU! - testing cpu vs cpu optimizer calculations
import time
tf.reset_default_graph()



new_saver = tf.train.import_meta_graph('./graph/final.meta')

with tf.Session() as sess:
  new_saver.restore(sess, tf.train.latest_checkpoint('./tmp'))

  optimize = tf.get_default_graph().get_operation_by_name( "optimizer" )
  c_loss = sess.run('loss:0', feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})
  print('initial:  c_loss', c_loss)  

  sess.run(optimize, feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})
  c_loss = sess.run('loss:0', feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})

  print('post:  c_loss', c_loss)

Output:
initial: c_loss 0.4687191
post: c_loss 0.5455321

#WITH GPU! - testing cpu vs gpu optimizer calculations
import time
tf.reset_default_graph()


new_saver = tf.train.import_meta_graph('gdrive/My Drive/graph/final.meta')

with tf.Session() as sess:
  new_saver.restore(sess, tf.train.latest_checkpoint('gdrive/My Drive/tmp'))

  optimize = tf.get_default_graph().get_operation_by_name( "optimizer" )
  c_loss = sess.run('loss:0', feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})
  print('initial:  c_loss', c_loss)  

  sess.run(optimize, feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})
  c_loss = sess.run('loss:0', feed_dict={'x:0': x_train[0:128], 'y:0': y_train[0:128], 'is_training:0': True})

  print('post:  c_loss', c_loss)

Output:
initial: c_loss 0.46873742
post: c_loss 0.5432756

EDIT: Also. Want to add I loaded the model with two different CPU sessions and found that the loss calculations above were identical. They only vary when I calculate the loss with GPU.

Answer 1

They might be different if you're shuffling the data you train the model on. Try fixing a numpy.random.seed(123) above your code (but after your imports) then disable shuffling and the losses should be identical.

For example, the tensorlayers mini-batch generator allows you to set the shuffle argument to False.

import numpy as np
import tensorlayers as tl

X = np.asarray([['a','a'], ['b','b'], ['c','c'], ['d','d'], ['e','e'], ['f','f']])
y = np.asarray([0,1,2,3,4,5])

for batch in tl.iterate.minibatches(inputs=X, targets=y, batch_size=16, shuffle=False):
    print(batch)