Results of training a Keras model different on Google Cloud

Question

I've created a script to train a keras neural net and have run it successfully on my machine (at the end of training there is roughly 0.8 validation accuracy). However, when I try to run the exact same code (on the same data) on a Google Cloud VM instance I get drastically worse results (~0.2 validation accuracy).

Git status confirms that the repo in the VM is up to date with master (same with my local machine), and I have verified that its versions of tf and keras are up to date (and same as my local machine). I've also set the numpy and tensorflow random seeds before importing Keras.

Has anyone run into a problem like this before? I'm at a loss for what could be causing this... the only difference I can think of is that my machine is running Python 3.6 whereas the VM is running Python 2.7. Could that account for the vast difference is training results?

Answer 1

I found a buggy interaction between Keras and the Estimator API in tensorflow 1.10 (current gcloud version), but not in >=1.11 (what I was using locally).

Not sure if it applies to you (do you use Keras+Estimator and tensorflow >=1.11 for local?)

I filed a bug report here: https://github.com/tensorflow/tensorflow/issues/24299