I am trying to implement L2 regularization to my data I wrote my function like this
def logicalregP3(xtr,ytr,learning_rate,iteration,lamda):
m=xtrain.T.shape[1]
n=xtrain.T.shape[0]
W= np.zeros((n,1))
B = 0
cost_list = []
for i in range (iteration):
z= np.array(np.dot(W.T, xtr.T),dtype=np.float32)+B
a= 1/(1 + np.exp(-z))
cost=-(1/m)*np.sum(ytr.T*np.log(a)+(1-ytr.T)*np.log(1-a))+(lamda*np.sum(W))
# Gradient Descent
regular=(lamda/(2*m))*W
dW=(1/m)*np.dot(a-ytr.T,xtr)+regular
dB=(1/m)*np.sum(a-ytr.T)
W=W-learning_rate*dW.T
B=B-learning_rate*dB
print("cost ", i ," ", cost)
cost_list.append(cost)
return W,B,cost_list
Then I call the function with these values
iterations=2000
learning_rate=0.0015
lamda=0.00001
Wp3,Bp3,cost_listp3=logicalregP3(xtrain,ytrain,learning_rate, iterations,lamda)
the output of my code:
cost 0 0.6931471824645996
cost 1 2.6271848720002735
cost 2 2.5287339955351484
cost 3 2.461344902235327
cost 4 2.4144645825946607
cost 5 2.3812474056254485
cost 6 2.357244340813794
cost 7 2.339536539714837
cost 8 2.3261846933514745
cost 9 2.3158827015378978
cost 10 2.307739196600699
cost 11 2.3011379381265
cost 12 2.2956485144978704
cost 13 2.290966932838124
cost 14 2.28687647957537
when I draw the graph: this is the output for the cost:
Is the output of my code normal or I have problem at the gradient descent
Actually L2 regularisation is not lambda * np.sum(W)
but it is lambda * np.linalg.norm(W, ord=2)
. In your case I assume that the gradient descent works well but you can't check it because your costs don't represent the quantity which is minimized here.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.