ValueError: Length of values (959) does not match length of index (5)

Question

Please help me out with below error. Tried Various post on stackoverflow, still not able to figure it out. It is throwing out Value error for 2nd iteration of loop even though shape and dataframes are intact and in right shape.

Thanks in Advance.

Please find the link of the Jupyter notebook here

def compute_cost(X,Y,W,b,lambda_=0):
    
    m = Y.shape[0]
    print(X.shape,W.shape)
    
    Z = np.dot(X,W) + b
    A = sigmoid(Z)
    J = (1/m) * np.sum((-Y * np.log(A) - (1 - Y) * np.log(1 - A)).values)
    
    return J.item(), A

def gradient_descent(X,Y,alpha=0.1,num_iters=100,lambda_=0):
    
    m = Y.shape[0]
    W = initialiaze_weights(X,Y) 
    b = 1
    
    for i in range(num_iters): 
        print('loop'+str(i))
        J, A = compute_cost(X,Y,W,b,lambda_)
        grad_W = (1/m) * np.dot(X.T,(A - Y)) # 959x11-959x5
        grad_b = (1/m) * np.sum(A - Y)
        #print(grad_W)
        W = W - alpha * grad_W
        b = b - alpha * grad_b
    
    return J

gradient_descent(X_train_norm,Y_train_dum)
loop0
(959, 11) (11, 5)
loop1
(959, 11) (11, 5)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-390-465bed6a5594> in <module>
----> 1 gradient_descent(X_train_norm,Y_train_dum)

<ipython-input-378-286b8d84d0b7> in gradient_descent(X, Y, alpha, num_iters, lambda_)
      7     for i in range(num_iters):
      8         print('loop'+str(i))
----> 9         J, A = compute_cost(X,Y,W,b,lambda_)
     10         grad_W = (1/m) * np.dot(X.T,(A - Y)) # 959x11-959x5
     11         grad_b = (1/m) * np.sum(A - Y)

<ipython-input-387-a8b943dff921> in compute_cost(X, Y, W, b, lambda_)
      4     print(X.shape,W.shape)
      5 
----> 6     Z = np.dot(X,W) + b
      7     A = sigmoid(Z)
      8     J = (1/m) * np.sum((-Y * np.log(A) - (1 - Y) * np.log(1 - A)).values)

c:\python38\lib\site-packages\pandas\core\generic.py in __array_ufunc__(self, ufunc, method, *inputs, **kwargs)
   2030         self, ufunc: np.ufunc, method: str, *inputs: Any, **kwargs: Any
   2031     ):
-> 2032         return arraylike.array_ufunc(self, ufunc, method, *inputs, **kwargs)
   2033 
   2034     # ideally we would define this to avoid the getattr checks, but
---skipped----some---part--of---this----error--
c:\python38\lib\site-packages\pandas\core\common.py in require_length_match(data, index)
    529     """
    530     if len(data) != len(index):
--> 531         raise ValueError(
    532             "Length of values "
    533             f"({len(data)}) "

ValueError: Length of values (959) does not match length of index (5)

Answer 1

This error gets solved if I convert dataframe into numpy before sending it to gradient descent.

gradient_descent(X_train_norm.to_numpy(),Y_train_dum.to_numpy())

But still in search of why the error occcured.