为什么在使用 TensorFlow 进行多元线性回归时会得到不同的权重？

Question

I have two implementations of multiple linear regressions, one using tensorflow and one using only numpy . I generate a dummy set of data and I try to recover the weights I used, but although the numpy one returns the initial weights, the tensorflow one always returns different weights (which also sort of work)

The numpy implementation is here , and here's the TF implementation:

import numpy as np
import tensorflow as tf

x = np.array([[i, i + 10] for i in range(100)]).astype(np.float32)
y = np.array([i * 0.4 + j * 0.9 + 1 for i, j in x]).astype(np.float32)

# Add bias
x = np.hstack((x, np.ones((x.shape[0], 1)))).astype(np.float32)

# Create variable for weights
n_features = x.shape[1]
np.random.rand(n_features)
w = tf.Variable(tf.random_normal([n_features, 1]))
w = tf.Print(w, [w])

# Loss function
y_hat = tf.matmul(x, w)
loss = tf.reduce_mean(tf.square(tf.sub(y, y_hat)))

operation = tf.train.GradientDescentOptimizer(learning_rate=0.000001).minimize(loss)

with tf.Session() as session:
    session.run(tf.initialize_all_variables())
    for iteration in range(5000):
        session.run(operation)
    weights = w.eval()
    print(weights)

Running the script gets me weights around [-0.481, 1.403, 0.701] , while running the numpy version gets me weights around [0.392, 0.907, 0.9288] which are much closer to the weights I used to generate the data: [0.4, 0.9, 1]

Both learning rates/epochs parameters are the same, and both initialise weights randomly. I don't normalize the data for either of the implementations, and I've ran them multiple times.

Why are the results different? I also tried to initialise weights in the TF version using w = tf.Variable(np.random.rand(n_features).reshape(n_features,1).astype(np.float32)) but that didn't fix it either. Is there something wrong with the TF implementation?

Answer 1

The problem appears to be with broadcasting. The shape of y_hat in the above is (100,1) , while y is (100,) . So, when you do tf.sub(y, y_hat) you end up with a matrix of (100,100) which are all the possible combinations of subtractions between the two vectors. I don't know, but I guess that you managed to avoid this in the numpy code.

Two ways to fix your code:

y = np.array([[i * 0.4 + j * 0.9 + 1 for i, j in x]]).astype(np.float32).T

or

y_hat = tf.squeeze(tf.matmul(x, w))

Although, that said, when I run this it still doesn't actually converge to the answer you want, but at least it's actually able to minimize the loss function.