Parameters Estimation using Least Square Method in Matlab

Question

I have the next questions: Consider a set of equations y=ax+b where i know the y and x and want to estimate the a and b using least square method. Let's assume to have Y=[y1 ; y2] Y=[y1 ; y2] and
A=[x1 1; x2 1] A=[x1 1; x2 1] so that Y=A*[a;b]

According to least square method : B=[a;b]=( transpose(A)*A )^-1*transpose(A)*Y

1. (A'*A) \\ A'*Y and A\\Y are the same?

2. Which is the best method to calculate B:

   inv( transpose(A)*A ) *transpose(A)*Y

   (transpose(A)*A) \\ transpose(A)*Y

   (A'*A) \\ A'*Y

   pinv(A)*Y (calculate the pseudo-inverse matrix)

all the above give slightly different results

Answer 1

Before solving your doubts, a remark is mandatory. When you want to transpose a matrix using the shorthand operator... you should not use ' , but .' . The first one is the shorthand operator for the conjugate transpose while the second one is the correct shorthand operator to use for the transpose . While they normally produce the same result, using the former with matrices containing complex numbers could mess up your calculations.

Since you didn't provide a data sample, here are the settings I deployed for my tests:

Y = [2; 4];
A = [3 1; 7 1];

Now, let's go step-by-step. For what concerns your first answer, yes, the two operations are equivalent on a mathematical point of view and produce basically the same result:

>> B = A \ Y

B =
    0.5
    0.5

-----------------------------

>> B = inv(A.' * A) * A.' * Y

B =
    0.500000000000001
    0.5

The slight difference you see is due to the fact that INV(A) * b is less accurate than A \\ b , and this is clearly stated even by the Matlab code interpreter if you hover a call to the inv function followed by a multiplication (which should be marked with an orange warning highlight):

This partially answers also your second question, but let's do an exhaustive benchmark. I discarded the computation performed using inv(A.' * A) * A.' * Yinv(A.' * A) * A.' * Y since it is recommended to avoid it. Here we go:

tic();
for i = 1:100000
    B = A \ Y;
end
toc();

tic();
for i = 1:100000
    B = pinv(A) * Y;
end
toc();

tic();
for i = 1:100000
    B = (A.' * A) \ A.' * Y;
end
toc();

This is the result of the benchmark:

Elapsed time is 0.187067 seconds.
Elapsed time is 2.987651 seconds.
Elapsed time is 2.173117 seconds.

Given that the three approaches have the same accurancy... the first one is by far, and obviously, the fastest one.