4.4 Application--OLS Estimation. Background.
avg of |di| = 10 = 10
Y-MZ is a vector =
Then S = || Y-MZ||2
S = || Y-MZ||2
Y and M are given since we have 3 data points to fit.
We simply need to select Z to minimize S.
Let P be the set of all vectors MZ where Z varies:
It turns out that all of the vectors in set P lie in the same plane through the origin (we discuss why later in the book).
The equation of the plane is
Take a=0,b=1, or a,b=0 and find that this plane contains:
And the normal vector will be U x V =
Recall that we are trying to minimize S = || Y-MZ||2
Y = (y1,y2,y3) is a point in space, and MZ is some vector in the set P which we have illustrated as a plane.
S = || Y-MZ||2 is the squared distance from the point to the plane, so if we can find the point,MA, in the plane closest to Y, we will have our solution.
Y-MA is orthogonal to all vectors,MZ, in the plane, so
(MZ) • (Y-MA) = 0
Note this rule for dot products when vectors are written as matrices:
0 = (MZ) • (Y-MA) =(MZ)T(Y-MA)=ZTMT(Y-MA)
=ZT(MTY-MTMA) = Z • (MTY-MTMA)
The last dot product is in two dimensions and tells that (MTY-MTMA) is orthogonal to every possible Z which can only happen if (MTY-MTMA) = 0,so
MTY=MTMA called the normal equations for A
With x1, x2,x3 all distinct, we can show that MTM is invertible, so from MTY=MTMA ,we get A = (MTM)-1MTY,
This will give us A=(a,b) which will give then give us the point (a+bx1,a+bx2,a+bx3) closest to Y.
Thus the best fit line will then be y=a + bx.
Recall that this argument started by defining n=3 so that we could use a 3 dimensional argument with vectors. The argument becomes more complex, but does extend to any n.
Then, the least squares approximating line has equation y=a0 + a1x where A = is found by Gaussian
elimination from the normal equations MTY=MTMA
Since at least two x’s are distinct, MTM is invertible so A=(MTM)-1MTY
We can generalize to select the least squares approximating polynmial of degree m: f(x)=a0+a1x+a2x2+…+anxn where we estimate the a’s
If n data points are given with at least m+1 x’s distinct, then
Then least squares approximating polynomial of degree m is: f(x)=a0+a1x+a2x2+…+anxn where
Is found by Gaussian elim from normal equations MTY=MTMA
Since at least m+1 x’s are distinct, MTM is invertible so