Machine Learning accuracy shows 0

Question

I need help with the accuracy for training the model for machine learning.

My input for the training are several array with 500 integer/data which I saved it in hdf5 file under dataset named 'the_data'. In this example, I have 100 arrays.

[[1,2,3,...500],
 [501,502,...1000],
 [1001,...       ],
 ....
 ......          ]]

The output is a random number which I generate before hand and save it as 'output.txt'. It has 100 random numbers.

194521, 307329, 182440, 180444, 275690,...,350879

Below is my modified script based on http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/

import h5py
from keras.models import Sequential
from keras.layers import Dense

seed = 7
np.random.seed(seed)

input_data = h5py.File('test.h5', 'r')
output_data = open("output.txt", "r")

X = input_data['the_data'][:]
Y = output_data.read().split(',')

model = Sequential()
model.add(Dense(500, input_dim=500, init='normal', activation='relu'))
model.add(Dense(100, init='normal', activation='relu'))
model.add(Dense(60, init='normal', activation='relu'))
model.add(Dense(1, init='normal', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adamax', metrics=['accuracy'])
model.fit(X, Y, nb_epoch=500, batch_size=10)
scores = model.evaluate(X, Y)

print("%s: %.2f%% , %s: %.2f%%" % (model.metrics_names[0], scores[0]*100, model.metrics_names[1], scores[1]*100))

What I got as the result is like this

Epoch 500/500
100/100 [==============================] - 0s - loss: -4851446.0896 - acc: 0.0000e+00
100/100 [==============================] - 0s
loss: -485144614.93% , acc: 0.00%

Does anyone have any thought about why does this happened?

Thank you for your help.

Answer 1

Do you know what the binary crossentropy is?

It's a loss function made for targets being binary (0 or 1). The loss is then some logarithm of the output or the output - 1 depending on the target value. So you can not apply it in your case.

You want to predict numerical values, so you should use something like root mean squarre errors.

The accuracy doesnt make sens either as you're not trying to predict a class but a real number value. It will rarely predict exactly the good one. Accuracy is used for example with binary crossentropy, then we can classify an output that is 0.7 as being in class 1. Or 0.2 being in class 0.

One more comment : why would you want to predict random values? It cant work... the network needs to recognize patterns and there is no patterns in random targets.

I hope this helps you a bit.

Answer 2

I agree with Nassim Ben. Try to use this

model.compile(loss='mean_square', optimizer='sgd')

Then, to calculate the accuracy you need a different way:

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(Y,Y_predicted)

print('MSE : %.3f' % mse)
print("Acc = ", 1-numpy.sqrt(mse))

This worked for me. But honestly, I feel like keras is not working good with predicting high numbers (other then between 0 and 1) I would be happy if I was wrong about this.