Keras Logistic回归在第一个时期返回nan

Question

I have a really simple data set. I cleaned the data (one hot encoding, normalizing the data and check for missing values or NaNs) and my learning rate is pretty small. But when tried to run a simple logistic regression using Keras and Theano as backend

model = Sequential() 
model.add(Dense(input_dim=84, activation='softmax',
            bias_initializer='normal', units=6)) 
rms = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)
model.compile(optimizer=rms, loss='categorical_crossentropy')


batch_size = 100
nb_epoch = 1
n = X.shape[0] # number of training examples
history = model.fit(X, Y_oh, batch_size=batch_size, epochs=nb_epoch)

Error:

Epoch 1/1
5459/5459 [==============================] - 0s - loss: nan

I checked here and tried to downgrade Theano to the version mentioned but it still gives the same error

Here is what X looks like

[[ 0.35755179  0.13747887  0.3        ...,  0.          0.          0.        ]
 [ 0.36401758  0.14963742  0.55       ...,  0.          0.          0.        ]
 [ 0.37889517  0.13775149  0.275      ...,  0.          0.          0.        ]
 ..., 
 [ 0.34387947  0.18706723  0.05       ...,  0.          0.          0.        ]
 [ 0.35708726  0.12905512  0.75       ...,  0.          0.          0.        ]
 [ 0.37915882  0.08061174  0.05       ...,  0.          1.          0.        ]]

and Y_oh (generated using the following code):

Y_oh = np_utils.to_categorical(Y.T[0],6)

[[ 0.  0.  0.  0.  0.  1.]
 [ 0.  0.  0.  0.  0.  1.]
 [ 0.  0.  0.  0.  0.  1.]
 ..., 
 [ 1.  0.  0.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.  0.]]

Answer 1

The problem is your activation here

model.add(Dense(input_dim=84, activation='softmax',
            bias_initializer='normal', units=6)) 

Swap 'softmax' for 'sigmoid' (or 'tanh') and you should be good.

Softmax has the property that the sum of its outputs is 1 , so that the network's output has a probability interpretation. The problem is, since you have only 1 output, it would either will never train (since the output will always be the same) or get unreasonable gradients trying to do so.

eg Fix

model.add(Dense(input_dim=84, activation='sigmoid',
            bias_initializer='normal', units=6)) 

Answer 2

Try to apply Log(0) and see what is the result. it is undefined "NAN" I answered this question in details here NAN in neural network