为什么 TensorFlow estimator 无法进行这种简单的线性回归预测

Question

I am learning tensorflow currently and cannot wrap my head around why tensorflow doesn't do proper prediction on following simple regression problem.

X are random numbers from 1000 to 8000 Y is X + 250

So if X is 2000, Y is 2250. This seems like a linear regression problem to me. Yet, when I try making a predictions, it is nowhere near close to what I would expect, X of 1000 is giving me prediction of 1048 instead of 1250.

Also loss and average loss are huge:

{'average_loss': 10269.81, 'loss': 82158.48, 'global_step': 1000}

Here is the complete code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
from sklearn.model_selection import train_test_split

x_data = np.random.randint(1000, 8000, 1000000)
y_true = x_data + 250

feat_cols = [tf.feature_column.numeric_column('x', shape=[1])]
estimator = tf.estimator.LinearRegressor(feature_columns=feat_cols)

x_train, x_eval, y_train, y_eval = train_test_split(x_data, y_true, test_size=0.3, random_state=101)

input_func = tf.estimator.inputs.numpy_input_fn({'x': x_train}, y_train, batch_size=8, num_epochs=None, shuffle=True)
train_input_func = tf.estimator.inputs.numpy_input_fn({'x': x_train}, y_train, batch_size=8, num_epochs=1000, shuffle=False)
eval_input_func = tf.estimator.inputs.numpy_input_fn({'x': x_eval}, y_eval, batch_size=8, num_epochs=1000, shuffle=False)

estimator.train(input_fn=input_func, steps=1000)

train_metrics = estimator.evaluate(input_fn=train_input_func, steps=1000)
eval_metrics = estimator.evaluate(input_fn=eval_input_func, steps=1000)

print(train_metrics)
print(eval_metrics)

brand_new_data = np.array([1000, 2000, 7000])
input_fn_predict = tf.estimator.inputs.numpy_input_fn({'x': brand_new_data}, shuffle=False)

prediction_result = estimator.predict(input_fn=input_fn_predict)

print(list(prediction_result))

Am I doing something wrong or am I misinterpreting what LinearRegression means?

Answer 1

i think it does when you tune some hyperparameters. I also changed the optimizer to AdamOptimizer .

Mainly the batch size is 1 and epochs is None .

train_input_func = tf.estimator.inputs.numpy_input_fn({'x': x_train}, y_train, batch_size=1, num_epochs=None, shuffle=True)

Code :

import tensorflow as tf
import numpy as np
from sklearn.model_selection import train_test_split

x_data = np.random.randint(1000, 8000, 10000)
y_true = x_data + 250


feat_cols = tf.feature_column.numeric_column('x')
optimizer = tf.train.AdamOptimizer(1e-3)

estimator = tf.estimator.LinearRegressor(feature_columns=[feat_cols],optimizer=optimizer)

x_train, x_eval, y_train, y_eval = train_test_split(x_data, y_true, test_size=0.3, random_state=101)


train_input_func = tf.estimator.inputs.numpy_input_fn({'x': x_train}, y_train, batch_size=1, num_epochs=None,
                                                      shuffle=True)

eval_input_func = tf.estimator.inputs.numpy_input_fn({'x': x_eval}, y_eval, batch_size=1, num_epochs=None,
                                                     shuffle=True)

estimator.train(input_fn=train_input_func, steps=1005555)

train_metrics = estimator.evaluate(input_fn=train_input_func, steps=10000)
eval_metrics = estimator.evaluate(input_fn=eval_input_func, steps=10000)

print(train_metrics)
print(eval_metrics)

brand_new_data = np.array([1000, 2000, 7000])
input_fn_predict = tf.estimator.inputs.numpy_input_fn({'x': brand_new_data}, num_epochs=1,shuffle=False)

prediction_result = estimator.predict(input_fn=input_fn_predict)

for prediction in prediction_result:
    print(prediction['predictions'])

Metrics :

{'average_loss': 3.9024353e-06, 'loss': 3.9024353e-06, 'global_step': 1005555}

{'average_loss': 3.9721594e-06, 'loss': 3.9721594e-06, 'global_step': 1005555}

[1250.003]

[2250.002]

[7249.997]

Answer 2

The reason of this slow convergence (you need to train for 1mln. steps, which is weird for such a seeming trivial problem) is that the data is not normalized.

Applying normalization I can train the model to predict the values accurately in 420 steps (I chose 420 because it's the number of memes) to get theses predictions:

[1250.387]
[2250.2046]
[7249.2915]

Code (done in TF 2.2):

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
from sklearn.model_selection import train_test_split

x_data = np.random.randint(1000, 8000, 1000000)
y_true = x_data + 250

tf.get_logger().setLevel('ERROR')

feat_cols = [tf.feature_column.numeric_column('size', shape=[1])]
estimator = tf.estimator.LinearRegressor(feature_columns=feat_cols, optimizer=tf.keras.optimizers.Adam(learning_rate=0.01))

def normalize(arr):
    return (arr - arr.mean()) / arr.std()

x_data_norm = normalize(x_data)
y_true_norm = normalize(y_true)

x_train, x_eval, y_train, y_eval = train_test_split(x_data_norm, y_true_norm, test_size=0.3, random_state=101)

# numpy_input_fn is for when you have the full dataset available in an array already and want a quick way to do batching/shuffling/repeating
input_func = tf.compat.v1.estimator.inputs.numpy_input_fn({'size': x_train}, y_train, batch_size=1, num_epochs=None, shuffle=True)
train_input_func = tf.compat.v1.estimator.inputs.numpy_input_fn({'size': x_train}, y_train, batch_size=1, num_epochs=None, shuffle=True)
eval_input_func = tf.compat.v1.estimator.inputs.numpy_input_fn({'size': x_eval}, y_eval, batch_size=1, num_epochs=None, shuffle=True)

estimator.train(input_fn=input_func, steps=420)

train_metrics = estimator.evaluate(input_fn=train_input_func, steps=500)
eval_metrics = estimator.evaluate(input_fn=eval_input_func, steps=500)

print(train_metrics)
print(eval_metrics)

# brand_new_data = np.array([1000, 2000, 7000])
# input_fn_predict = tf.compat.v1.estimator.inputs.numpy_input_fn({'size': brand_new_data}, num_epochs=1, shuffle=False)
# prediction_result = estimator.predict(input_fn=input_fn_predict)
# print(list(prediction_result))

# predict w/ normalization
d1 = 1000 
d2 = 2000
d3 = 7000
brand_new_data = np.array([(d1 - x_data.mean()) / x_data.std(), (d2 - x_data.mean()) / x_data.std(), (d3 - x_data.mean()) / x_data.std()])

input_fn_predict = tf.compat.v1.estimator.inputs.numpy_input_fn({'size': brand_new_data}, num_epochs=1, shuffle=False)
prediction_result = estimator.predict(input_fn=input_fn_predict)

for res in prediction_result:
  print(res['predictions'] * y_true.std() + y_true.mean())