在 Numpy Python 中从头开始进行正交线性回归的梯度下降推导

Question

I wanted to derive gradient descent from scratch using the error function for Orthogonal linear regression. So that I could write python code in NumPy. However, I am unsure about the error function in this.

y = W0 + W1 * X

Can anyone help me derive this!! Thanks in advance.

Answer 1

Let's say you have y = m*x + p with (m; p) being the parameters you are tuning.

When you feed an observation sample X of X' through your model, you obtain a vector Y'. Then, you can compute your loss, say a mean square error, as e = 1 / n * sum_i (Yi - Yi')²' where Y is the y-coordinates associated to the x-coordinates in X .

Now, we want to differentiate e relative to your parameters:

• de / dm = -2 / sigma² * sum_i ( Xi' * ( Yi - p - m * Xi' ) )
• de / dp = -2 / sigma² * sum_i ( Yi - p - m * Xi' )

Then you want to minimize e thanks to this gradient:

• m <- m - lr * de / dm
• p <- p - lr * de / dp

With lr being your learning rate.

Do the math on your own though, just in case I made an error.

EDIT: I've implemented it quickly using those formulas and the math seems ok (at least it converges to the solution quickly)