How to get reproducible results using tensorflow-gpu in CNN model?

Question

I'm using Tensorflow GPU in CNN model and I would like to reproduce my results on each execution.

I have tried setting the graph level seed, numpy random seed, and even operation level seeds. However, it still not reproducible.

I have also tried Tensorflow with CPU; I have got a reproducible result, but it's so slow.

Any idea or debugging experiment suggestion to get a reproducible result using Tensorflow with GPU?

Answer 1

Did you try to save the weights of your model at each iteration ? Using keras callbacks for exemple :

filepath = 'weights/' + model.name + '.{epoch:02d}-{acc:.2f}.hdf5'
checkpoint = ModelCheckpoint(filepath, monitor='acc', verbose=0, save_weights_only=True, save_best_only=True, mode='auto', period=1)

You can then load the weights of your model at the iteration you want

Hope i understood well you problem !