每次使用TensorFlow训练CNN（MNIST数据集）时，如何获得相同的损失值？

Question

I want to train a convolutional neural network (with MNIST data set and TensorFlow) a few times new and get every time the same results of the accuracy. To get this i:

1. Save an untrained only initialized (global_variables_initializer) net
2. Load every time I start the training this untrained net
3. Set mnist.train.next_batch shuffle=False, so the image sequence is every time the same

I have done this before with a feed forward net (3 hidden layer) and every time I run this python script I get the exact same values for loss and accuracy.

But, the "same" script with changing the model from a feed forward net to a convolutional neural net make every time I run the script a little different loss/accuracy.

So I reduce the batch size to one and look for each image the loss value and see that the first two images always have the same loss value, but the rest is every time I run the script a little different.

Any idea why?

Answer 1

Thanks to @AlexandrePassos comment, I search for deterministic/ non-deterministic operations in TensorFlow.

So at the moment all operations which using CUDA atomics and running on GPU, are non-deterministic.
See this link: https://github.com/tensorflow/tensorflow/issues/3103

If somebody knows a way to realize a CNN with TensorFlow on GPU and with deterministic operations, please: How to create a CNN with deterministic operations in TensorFlow on a GPU?