Tensorflow Keras：在 model.fit() 上进行训练

Question

I am new to tensorflow and python, so I'm sure its just something stupid. But when I try to train my model with model.fit(), after a random number of training iterations it freezes. GPU usage goes to zero, and no error message is shown to indicate where the error is. The model I am trying to train is as follows:

SizeOfInput = 50
VocabSize = 8000
EmbeddingSize = 200

model = Sequential()
model.add(Embedding(input_dim=VocabSize, output_dim=EmbeddingSize, input_length=SizeOfInput))
model.add(LSTM(256, input_shape=(SizeOfInput, EmbeddingSize), return_sequences=True))
model.add(Dropout(0.1))
model.add(LSTM(256))
model.add(Dropout(0.1))
model.add(Dense(VocabSize, activation='softmax'))

optimizer = tf.keras.optimizers.RMSprop(lr=1e-3, decay=1e-5)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)

Tensorflow: 2.2.0

Python: 3.8

GPU: RTX2080Ti

I am training on 10000 randomly sampled sequences of 50 integers each epoch, from a larger dataset:

model.fit(x, y, batch_size=100, epochs=1)

I really don't know what the issue is, let me know if you need more information

Answer 1

1.Firstly please try your code in Google Colab

2.Try to use another optimizer:

keras.layers.BatchNormalization._USE_V2_BEHAVIOR = False

3.Maybe some ops running in cpu and some Ops are running in GPU. In order to prevent it:

with tf.device("/device:GPU:0"):
    model.fit(x, y, batch_size=100, epochs=1)

3.Update the latest version of

tf-nightly-gpu 
tf-gpu 2.1 cudn 10.2

4.Test this:

print(tf.version.VERSION, tf.executing_eagerly(), keras.layers.BatchNormalization._USE_V2_BEHAVIOR)

keras.layers.BatchNormalization._USE_V2_BEHAVIOR = False