Keras Model With CuDNNLSTM Layers Doesn't Work on Production Server

Question

I have used an AWS p3 instance to train the following model using GPU acceleration:

x = CuDNNLSTM(128, return_sequences=True)(inputs)
x = Dropout(0.2)(x)
x = CuDNNLSTM(128, return_sequences=False)(x)
x = Dropout(0.2)(x)
predictions = Dense(1, activation='tanh')(x)
model = Model(inputs=inputs, outputs=predictions)

After training I saved the model with Keras' save_model function and moved it to a separate production server that doesn't have a GPU.

When I attempt to predict using the model on the production server it fails with the following error:

No OpKernel was registered to support Op 'CudnnRNN' with these attrs. Registered devices: [CPU], Registered kernels:

I'm guessing this is because the production server doesn't have GPU support, but I was hoping this wouldn't be a problem. Is there any way I can use this model on a production server without a GPU?

Answer 1

No, you can't, CuDNN requires the use of a CUDA GPU. You have to replace your CuDNNLSTM layers with standard LSTM ones.

Answer 2

short answer: Yes, you can.

Just need to re create your architecture with LSTM layer instead of CuDNNLSTM.

Your code should be the following:

x = LSTM(128, return_sequences=True, recurrent_activation='sigmoid')(inputs)
x = Dropout(0.2)(x)
x = LSTM(128, return_sequences=False, recurrent_activation='sigmoid')(x)
x = Dropout(0.2)(x)
predictions = Dense(1, activation='tanh')(x)
model = Model(inputs=inputs, outputs=predictions)

and then

model.load_weights(path_to_your_weights_file)

Notice the recurrent_activation='sigmoid' . This is very important.

Long explanation:

LSTM and CuDNNLSTM are compatible eacth other, so you can load the weigths from one to another with no problem. However, their default values for the activation function are slightly different. Some times it leads to small differences between one and the other, but it may lead to very big ones, as reported here .

[The activation values] for CuDNNLSTM [...] are hard-coded in CuDNN and cannot be changed from Keras. They correspond to activation='tanh' and recurrent_activation='sigmoid' (slightly different than default hard_sigmoid in [LSTM] Keras). ref

Answer 3

The easiest way to solve this would be to Replace the CuDNN layers with regular keras layer, ie. convert CudNNLSTM to LSTM ,etc

If using Google Colab go to Runtime>Change Runtime Settings and set the accelerator to GPU

Answer 4

You definitely can train on CuDNNLSTM, then run inference on LSTM.

The trick is to change your layer architecture from CuDNNLSTM to LSTM in your .json file, before loading your h5 file. The 2x bias in the CuDNNLSTM weights will automatically convert when you load your h5 file, but Keras won't automatically change your .json file for you.

In other words, just open up your saved .json model, change all instances of CuDNNLSTM to LSTM, save the .json file, then load your .h5 file. You should then be able to run inference with the model.

Answer 5

试试

pip install tensorflow-gpu