Twitter Sentiment Analysis Constant Zero (0.0000e+00) Loss value

Question

I'm trying to figure out why model's Loss value is always 0.0, so the accuracy seems to be constant as well (which is incorrect in my case, afaik).

Code snippet:

model = Sequential()
model.add(Embedding(vocab_size, glove_vectors.vector_size, weights=[embedding_matrix], input_length=X.shape[1]))
model.add(Dropout(0.5))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation="sigmoid"))
model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=["accuracy"])
model.summary()

EPOCHS = 20

train_data, test_data, train_labels, test_labels = train_test_split(X, Y, test_size=0.20, random_state = 42)


print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)



val_data = (test_data, test_labels)
history = model.fit(train_data, train_labels, validation_data=val_data, epochs=EPOCHS)
score = model.evaluate(test_data, test_labels)

Output:

Epoch 1/20
25/25 [==============================] - 4s 69ms/step - loss: 0.0000e+00 - accuracy: 0.5241 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 2/20
25/25 [==============================] - 1s 55ms/step - loss: 0.0000e+00 - accuracy: 0.4927 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 3/20
25/25 [==============================] - 1s 55ms/step - loss: 0.0000e+00 - accuracy: 0.5110 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 4/20
25/25 [==============================] - 1s 56ms/step - loss: 0.0000e+00 - accuracy: 0.5074 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 5/20
25/25 [==============================] - 1s 55ms/step - loss: 0.0000e+00 - accuracy: 0.5363 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 6/20
25/25 [==============================] - 1s 53ms/step - loss: 0.0000e+00 - accuracy: 0.5042 - val_loss: 0.0000e+00 - val_accuracy: 0.4650
Epoch 7/20
25/25 [==============================] - 1s 54ms/step - loss: 0.0000e+00 - accuracy: 0.4904 - val_loss: 0.0000e+00 - val_accuracy: 0.4650

Answer 1

In binary classification there will be 1 node in the output layer even though we will be predicting between two classes. In order to get the output in a probability format between 0 and 1 we will use the sigmoid function.

Hence binary_crossentropy is the correct loss function in your case

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=["accuracy"])