这个梯度下降有什么问题？

Question

I want to minimize ||Ah(Wx)-y|| by gradient descent where h us ReLU

s = 9
n = 99
m = 999
A = np.random.normal((n,m))
y = np.random.normal((m,1))
W = np.random.normal((n,s))

def obj_fcn(x):
    return np.linalg.norm(A.dot(np.max(W.dot(x),0))-y)

if __name__ == '__main__':
    dx = 10
    max_iter = 99
    i = 0
    x = np.zeros((s,1))
    ddx = 0.00001
    ddx2 = ddx*2
    while i<max_iter:
        d_obj = (obj_fcn(x+ddx)-obj_fcn(x-ddx))/ddx2
        x = x + d_obj
        print(x)
        i = i+1

This gives error

/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/Chu/Documents/fun/m608.py
Traceback (most recent call last):
  File "/Users/Chu/Documents/fun/m608.py", line 21, in <module>
    d_obj = (obj_fcn(x+ddx)-obj_fcn(x-ddx))/ddx2
  File "/Users/Chu/Documents/fun/m608.py", line 11, in obj_fcn
    return np.linalg.norm(A.dot(np.max(W.dot(x),0))-y)
ValueError: shapes (2,) and (9,1) not aligned: 2 (dim 0) != 9 (dim 0)

What's wrong? Is there an book I can read about such issues in Python? I almost always have this kind of trouble.

Answer 1

I broke down your function to individual operations and printed the inputs and results as soon as computed. This is a very basic debugging technique; you need to learn this skill if you're going to program at any level beyond trivial applications. See this lovely debug blog for help.

def obj_fcn(x):
    print("x", x)
    print("W", W)
    wdot = W.dot(x)
    print("wdot", wdot)
    wdot = np.max(wdot, 0)
    adot = A.dot(wdot)
    print("adot", adot)
    norm_arg = adot - y   
    print("arg", norm_arg)
    result = np.linalg.norm(norm_arg)
    return result
    # return np.linalg.norm(A.dot(np.max(W.dot(x),0))-y)

Output:

x [[  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]
 [  1.00000000e-05]]
W [ 98.90267125   7.01706833]
Traceback (most recent call last):
  File "so.py", line 32, in <module>
    d_obj = (obj_fcn(x+ddx) - obj_fcn(x-ddx)) / ddx2
  File "so.py", line 13, in obj_fcn
    wdot = W.dot(x)
ValueError: shapes (2,) and (9,1) not aligned: 2 (dim 0) != 9 (dim 0)

That shows the error clearly: you're trying to take the dot product of items of different lengths. Dot product is not defined for such a case. You have to fix your input processing.