Keras始终输出恒定值

Question

I've been training a simple Artificial Neural Network using Keras

model = Sequential([
    Dense(32, input_shape=(32,), activation = 'relu'),
    Dense(20, activation='relu'),
    Dense(65, input_shape=(65,), activation='softmax')
])

model.summary()
model.compile(Adam(lr=.001), loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_samples, train_labels, batch_size=1000, epochs=10000,shuffle = True, verbose=2)

After training, I test the model, and it always outputs a constant value. What shall I do?

https://github.com/keras-team/keras/issues/1727

The link above tells that I have to center my data to have a zero mean. I don't have any idea to do that.

Answer 1

If you want to center your data to have zero mean, just subtract sample mean from each samples.

eg if you have features data and want to center it to have zero mean, you can do that using numpy's mean function as follows.

features = features - np.mean(features)

You may also need to normalize using standard deviation. This can be achieved by using numpy as follows.

normalized_features = (features - np.mean(features))/ np.std(features)

I hope this helps.

Answer 2

You're using 'relu' activations without caring for it going to all zeros.

Relu has a "zero region" when its input is negative, and this region, naturally has no gradient, so there is no possibility of it changing over training.

If all neurons in a layer go to the zero region, your model is frozen forever.

One thing you can do is to replace 'relu' by 'sigmoid' or 'tanh' .
Another thing is to use a BatchNormalization layer before it.

The BatchNormalization does the centering for you, besides adding some speed to training and a little regularization.

model = Sequential([

    #optional BatchNormalization(input_shape(32,))

    Dense(32, input_shape=(32,)),
    BatchNormalization(),
    Activation('relu')
    Dense(20),
    BatchNormalization(),
    Activation('relu'),
    Dense(65, input_shape=(65,), activation='softmax')
])