从零开始的逻辑回归

Question

I am implementing multinomial logistic regression using gradient descent + L2 regularization on the MNIST dataset. My training data is a dataframe with shape (n_samples=1198, features=65). On each iteration of gradient descent, I take a linear combination of the weights and inputs to obtain 1198 activations (beta^T * X). I then pass these activations through a softmax function. However, I am confused about how I would obtain a probability distribution over 10 output classes for each activation?

My weights are initialized as such

n_features = 65

# init random weights
beta = np.random.uniform(0, 1, n_features).reshape(1, -1)

This is my current implementation.

def softmax(x:np.ndarray): 
    exps = np.exp(x)
    return exps/np.sum(exps, axis=0)

def cross_entropy(y_hat:np.ndarray, y:np.ndarray, beta:np.ndarray) -> float:
    """
    Computes cross entropy for multiclass classification  
    y_hat: predicted classes, n_samples x n_feats
    y: ground truth classes, n_samples x 1
    """
    n = len(y)
    return - np.sum(y * np.log(y_hat) + beta**2 / n)

    
def gd(X:pd.DataFrame, y:pd.Series, beta:np.ndarray,
       lr:float, N:int, iterations:int) -> (np.ndarray,np.ndarray):
    
    """
    Gradient descent
    """
    n = len(y)
    cost_history = np.zeros(iterations)
    
    for it in range(iterations):
        
        activations = X.dot(beta.T).values
        
        y_hat = softmax(activations)
                
        cost_history[it] = cross_entropy(y_hat, y, beta)
                
        # gradient of weights
        grads = np.sum((y_hat - y) * X).values
        
        # update weights
        beta = beta - lr * (grads + 2/n * beta)
        
    return beta, cost_history

Answer 1

In Multinomial Logistic Regression, you need a separate set of parameters (the pixel weights in your case) for every class . The probability of an instance belonging to a certain class is then estimated as the softmax function of the instance's score for that class. The softmax function makes sure that the estimated probabilities sum to 1 over all classes.