Training RNN on GPU - which tf.keras layer should I use?

Question

I am training RNNs, which I built using tf.keras.layers.GRU layers. They are taking a long time to train (>2 hours), so I am going to deploy them to the GPU for training. I am wondering a few things about training on GPU:

1. What is the difference between tf.keras.layers.CuDNNGRU and tf.keras.layers.GRU (and also tf.keras.layers.LSTM vs. tf.keras.layers.CuDNNLSTM )? I understand from this post that CuDNNGRU layers train faster than GRU layers, but
   • Do the 2 layers converge to different results with the same seed?
   • Do the 2 layers perform the same during inference?
   • Do CuDNN layers require a GPU during inference?
   • Can GRU layers run inference on a GPU?
   • Are CuDNN layers easily deployable? I am currently using coremlconverter to convert my keras model to CoreML for deployment.
2. Is there an equivalent CuDNN layer for tf.keras.layers.SimpleRNN (ie tf.keras.layers.CuDNNSimpleRNN )? I am not committed to a specific architecture yet, and so I believe I would need the tf.keras.layers.CuDNNSimpleRNN layer if I decide on SimpleRNNs and the CuDNN layer has some functionality that I need.
3. With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?

Answer 1

if you are using a cuda compatible gpu, it makes absolutely sense to use CuDNN layers. They have a different implementation that tries to overcome computation parallelization issues inherent in the RNN architecture. They usually perform a bit worst though but are 3x-6x faster https://twitter.com/fchollet/status/918170264608817152?lang=en

Do the 2 layers converge to different results with the same seed?

yes

Do the 2 layers perform the same during inference?

You should have a comparable performance but not exactly the same

Do CuDNN layers require a GPU during inference?

Yes but you can convert to a CuDNN compatible GRU/LSTM

Can GRU layers run inference on a GPU?

Yes

With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?

Yes and you need a cuda compatible gpu