在更大的阵列上训练时的损失变为 inf 然后变为 nan(Tensorflow)

Question

this is probably the simplest model ever and I wrote this to demonstrate in a webinar that I will take in a few days

import numpy as np
from tensorflow import keras
model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')


num = []
sqr = []
for i in range(20):
  num.append(i)
  sqr.append(i*i)
  print(num[i], sqr[i])

def model():
    xs = np.array(num, dtype=float)
    ys = np.array(sqr, dtype=float)
    global model
    model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
    model.compile(optimizer='sgd', loss='mean_squared_error')
    model.fit(xs, ys, epochs=500)


model()

print(model.predict([10]))

As you can see it is just a NN to predict the Square of a number. but this gives a inf and then a nan as loss

1/1 [==============================] - 0s 2ms/step - loss: nan
Epoch 499/500
1/1 [==============================] - 0s 5ms/step - loss: nan
Epoch 500/500
1/1 [==============================] - 0s 1ms/step - loss: nan

the prediction gives [[nan]]

If I reduce the 20 to a 7 or 8, it works. but then fails with anything above that.

I think it has something to do with Learning rate, but I could be wrong... Please educate me about how this workd and a solution.

Answer 1

Yes, it's for the learning rate. Just set the learning rate to 0.001 and you are good to go:

import numpy as np
from tensorflow import keras
model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')


num = []
sqr = []
for i in range(20):
  num.append(i)
  sqr.append(i*i)
  print(num[i], sqr[i])

def model():
    xs = np.array(num, dtype=float)
    ys = np.array(sqr, dtype=float)
    global model
    model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
    opt = keras.optimizers.SGD(learning_rate = 0.001)
    model.compile(optimizer = opt, loss='mean_squared_error')
    model.fit(xs, ys, epochs=500)


model()

print(model.predict([10]))

or you can just change the loss function to mean_absolute_error or use different optimizers.

The reason: your numbers are very big and the mean_squared_error uses 2 * |y - pred| in gradient computation so the steps that the optimizer will take in each iteration is very big and it will diverge. so by multiplying it in a smaller number (0.001 instead of 0.01) we will help it to have smaller steps and converge.

Answer 2

It gives this kind of error when there is an overflow or division by zero. Normalize your input data and also try to reduce the learning rate.