Error: mat1 and mat2 shapes cannot be multiplied (1000x10 and 1x1)

Question

I am trying to implement Ridge Regression in pytorch, defining the loss function and plotting said function over different iterations. The only issue is, I keep getting an error code: mat1 and mat2 shapes cannot be multiplied (1000x10 and 1x1). I would like to convert the second matrix to a 1x10 in order to complete the code but I can't seem to get it to work.

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline  

n = 1000
p = 10

mean = np.zeros((p))
val = 0.8
cov = np.ones((p,p))*val
cov = cov + np.eye(p)*(1-val)

np.random.seed(10)
X = np.random.multivariate_normal(mean, cov, n)
theta_true = np.concatenate((np.ones((5,1)), np.zeros((5,1))),axis=0)

delta=0.5
Sigma = np.eye(n,n,k=-1)*0.4 + np.eye(n,n)*1 + np.eye(n,n,k=1)*0.4
mean = np.zeros(n)
e = np.random.multivariate_normal(mean, Sigma, 1)

y=X@theta_true + delta*e.T

import torch
X_t = torch.from_numpy(X).float()
y_t = torch.from_numpy(y).float()
Sigma_t = torch.from_numpy(Sigma).float()

import torch.nn as nn
import torch.nn.functional as F

class MyLinear(nn.Module):
    def __init__(self): 
        super(MyLinear, self).__init__()
        self.linear = nn.Linear(1, 1)  
    def forward(self, x): 
        out = self.linear(x)
        return out

def L2_norm(model):
    return torch.sum(list(model.parameters())[0]**2)    

def L1_norm(model):
    return torch.sum(torch.abs(list(model.parameters())[0]))

def ridge_loss(y_pred, y_true, model, lambda_):
    mse = F.mse_loss(y_pred, y_true)
    regularization = lambda_ * L2_norm(model)
    return mse + regularization

import matplotlib.pyplot as plt

model = MyLinear()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

lambda_ = 0.1
num_epochs = 1000
loss_values = []

for epoch in range(num_epochs):
    optimizer.zero_grad()
    y_pred = model(X_t)
    loss = ridge_loss(y_pred, y_t, model, lambda_)
    loss_values.append(loss.item())
    loss.backward()
    optimizer.step()

plt.plot(loss_values)
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.title('Ridge Regression Loss over Iterations')
plt.show()

I tried changing the theta_true definition to transform the matrix but the same error occurred.

theta_true = np.concatenate((np.ones((5,1)), np.zeros((5,1)))).reshape(10, 1)

Answer 1

Your Linear layer in MyLinear (line 37) is what is causing the issue.

self.linear = nn.Linear(1, 1)

means 1 input channel, one output channel, but x, as you have it here has shape (1000, 10), meaning it has 10 channels. So you will need to change that line to

self.linear = nn.Linear(10, 1)

that will do the trick, here is the image I get with that change: