Keras 神经网络为所有输入预测相同的数字

Question

I am trying to create a keras neural network to predict distance on roads between two points in city. I am using Google Maps to get travel distance and then train neural network to do that.

import pandas as pd
arr=[]
for i in range(0,100):
    arr.append(generateTwoPoints(55.901819,37.344735,55.589537,37.832254))
    df=pd.DataFrame(arr,columns=['p1Lat','p1Lon','p2Lat','p2Lon', 'distnaceInMeters', 'timeInSeconds'])
print(df)

Neural network architecture:

from keras.optimizers import SGD
sgd = SGD(lr=0.00000001)
from keras.models import Sequential
from keras.layers import Dense, Activation
model = Sequential()
model.add(Dense(100, input_dim=4 , activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(1,activation='sigmoid'))
model.compile(loss='mse', optimizer='sgd', metrics=['mse'])

Then i divide sets to test/train

Xtrain=train[['p1Lat','p1Lon','p2Lat','p2Lon']]/100
Ytrain=train[['distnaceInMeters']]/100000
Xtest=test[['p1Lat','p1Lon','p2Lat','p2Lon']]/100
Ytest=test[['distnaceInMeters']]/100000

Then i fit data into the model, but loss stays the same:

history = model.fit(Xtrain, Ytrain,
                    batch_size=1,
                    epochs=1000,
                    # We pass some validation for
                    # monitoring validation loss and metrics
                    # at the end of each epoch
                    validation_data=(Xtest, Ytest))

I later print the data:

prediction = model.predict(Xtest)
print(prediction)
print (Ytest)

But result is the same for all the inputs:

[[0.26150784]
 [0.26171574]
 [0.2617755 ]
 [0.2615582 ]
 [0.26173398]
 [0.26166356]
 [0.26185763]
 [0.26188275]
 [0.2614446 ]
 [0.2616575 ]
 [0.26175532]
 [0.2615183 ]
 [0.2618127 ]]
    distnaceInMeters
2            0.13595
6            0.27998
7            0.48849
16           0.36553
21           0.37910
22           0.40176
33           0.09173
39           0.24542
53           0.04216
55           0.38212
62           0.39972
64           0.29153
87           0.08788

I can not find the problem. What is it? I am new to machine learning.

Answer 1

You are doing a very elementary mistake: since you are in a regression setting, you should not use a sigmoid activation for your final layer (this is used for binary classification cases); change your last layer to

model.add(Dense(1,activation='linear'))

or even

model.add(Dense(1))

since, according to the docs , if you do not specify the activation argument it defaults to linear .

Various other advice offered already in the other answer and the comments may be useful (lower LR, more layers, other optimizers eg Adam ), and you certainly need to increase your batch size; but nothing will work with the sigmoid activation function you currently use for your last layer.

Irrelevant to the issue, but in regression settings you don't need to repeat your loss function as a metric; this

model.compile(loss='mse', optimizer='sgd')

will suffice.

Answer 2

It would be very useful if you could post the the progression of the loss and MSE (of both the training and validation/test set) as it goes throughout the training. Even better, it would be best if you can visualize it as per https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/ and post the vizualization here.

In the meantime, based on the facts: 1) You say the loss isn't decreasing (I'm assuming on the training set, during training, based on your compile args). 2) You say that the prediction "accuracy" on your test set is bad. 3) My experience/intuition (not an empirical assessment) tells me that your two layer dense model is a little too small to be able to capture the complexity inherent in your data. AKA your model is suffering from too high a Bias https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229

The fastest and easiest thing you can try, is to try to add both more layers and more nodes to each layer.

However, I should note that there is a lot of causal information that can affect the driving distance and driving time beyond just the the distance between two coordinates, which might be the feature that your Neural network will most readily extract. For example, whether you drive on a highway or sides treets, traffic lights, whehter the roads twist and turn or go straight... to infer all of that just from that the data you will need enormous amounts of data(examples) in my opinion. If you could add input columns with eg disatance to nearest higway from both points, you might be able to train with less data

I would also reccomend that you souble check that you are feeding as input what you think you are feeding (and its shape), and also, you should use some standardization from function sklearn which might help the model learn faster and converge faster to a higher "accuracy".

If and when you post either more code or the training history I can help you more (and also how many training samples).

EDIT 1: Try changing batch size to a larger number preferably batch_size=32 if it fits in your memory. you can use a small batch size (such as 1) when working with an "info rich" input like an image, but when using a very "info poor" datum like 4 floats (2 coordinates), the gradient will point each batch (with batch_size=1 ) to a practically random (pseudo...) direction and not neccessarily get any closer to a local minimum. Only when taking the gradient on the collective loss of a larger batch (like 32, and perhaps more) will you get a gradient that points at least approximately in the direction of the local minimum and converge to a better result. Also, I suggest that you don't mess with the learning rate manually and perhaps change to an optimizer like "adam" or "RMSProp".

Edit 2 : @Desertnaut made an excellent point that I totally missed, a correction without which, your code will not work properly. He deserves the credit so I will not include it here. Please refer to his answer. Also, don't forget to raise your batch size, and not "manually mess" with your learning rate, "adam" for example, will do it for you.