Keras Binary Model Getting stuck at 50% Accuracy

Question

I'm training a model to understand the impact of news on market volatility. The model seems to be find and the dataset classes are balanced, so I'm not sure what's exactly wrong.

I have coded a basic model using pretrained word embeddings:

model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, weights=[embedding_matrix]),
    tf.keras.layers.LSTM(300, return_sequences=True, activation='relu'),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(254, activation='relu')),
    tf.keras.layers.Dropout(0.4),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.4),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['binary_accuracy'])

Training the model, I get this:

109/109 [==============================] - 265s 2s/step - loss: 0.6945 - 
binary_accuracy: 0.5032 - val_loss: 0.6927 - val_binary_accuracy: 0.5161

109/109 [==============================] - 265s 2s/step - loss: 0.6945 - 
binary_accuracy: 0.5032 - val_loss: 0.6978 - val_binary_accuracy: 0.5123

109/109 [==============================] - 265s 2s/step - loss: 0.6945 - 
binary_accuracy: 0.5032 - val_loss: 0.6859 - val_binary_accuracy: 0.5096

109/109 [==============================] - 265s 2s/step - loss: 0.6945 - 
binary_accuracy: 0.5032 - val_loss: 0.6801 - val_binary_accuracy: 0.5245

I thought maybe my issue is that the data somehow isn't related, and the model has nothing to learn, but I'm not even sure about that, actually, I have published the dataset and the notebook on GitHub so that you can reproduce the issue, will be great if you can find what is going on.

Answer 1

Your model seems too complex for the size of your dataset, 5,000 examples. I would suggest the following:

model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, weights=[embedding_matrix]),
    tf.keras.layers.LSTM(128, return_sequences=True, activation='relu'),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128, activation='relu')),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(2, activation='sigmoid')
])

model.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['binary_accuracy'])

Note that you have to change the output layer to 2 neurons:

tf.keras.layers.Dense(2, activation='sigmoid')

You can do it by applying pd.get_dummies(y_train)

Also, reduce Dropout to 0.2 and work on learning rate. Don't forget to normalize data in the interval 0 to 1:

def norm(x):
    return (x-np.min(x))/(np.max(x)-np.min(x))