1 / 56

Ordinary Least-Squares

Ordinary Least-Squares. Outline. Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem. One-dimensional regression. One-dimensional regression. Find a line that represent the ”best” linear relationship:. One-dimensional regression.

arnon
Download Presentation

Ordinary Least-Squares

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ordinary Least-Squares Ordinary Least-Squares

  2. Outline • Linear regression • Geometry of least-squares • Discussion of the Gauss-Markov theorem Ordinary Least-Squares

  3. One-dimensional regression Ordinary Least-Squares

  4. One-dimensional regression Find a line that represent the ”best” linear relationship: Ordinary Least-Squares

  5. One-dimensional regression • Problem: the data does not go through a line Ordinary Least-Squares

  6. One-dimensional regression • Problem: the data does not go through a line • Find the line that minimizes the sum: Ordinary Least-Squares

  7. One-dimensional regression • Problem: the data does not go through a line • Find the line that minimizes the sum: • We are looking for that minimizes Ordinary Least-Squares

  8. Matrix notation Using the following notations and Ordinary Least-Squares

  9. Matrix notation Using the following notations and we can rewrite the error function using linear algebra as: Ordinary Least-Squares

  10. Matrix notation Using the following notations and we can rewrite the error function using linear algebra as: Ordinary Least-Squares

  11. Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

  12. Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

  13. Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

  14. Multidimentional linear regression Using a model with m parameters and n measurements Ordinary Least-Squares

  15. Multidimentional linear regression Using a model with m parameters and n measurements Ordinary Least-Squares

  16. Ordinary Least-Squares

  17. Ordinary Least-Squares

  18. parameter 1 Ordinary Least-Squares

  19. parameter 1 measurement n Ordinary Least-Squares

  20. Minimizing Ordinary Least-Squares

  21. Minimizing Ordinary Least-Squares

  22. Minimizing is flat at Ordinary Least-Squares

  23. Minimizing is flat at Ordinary Least-Squares

  24. Minimizing is flat at does not go down around Ordinary Least-Squares

  25. Minimizing is flat at does not go down around Ordinary Least-Squares

  26. Positive semi-definite In 1-D In 2-D Ordinary Least-Squares

  27. Minimizing Ordinary Least-Squares

  28. Minimizing Ordinary Least-Squares

  29. Minimizing Ordinary Least-Squares

  30. Minimizing Always true Ordinary Least-Squares

  31. Minimizing The normal equation Always true Ordinary Least-Squares

  32. Geometric interpretation Ordinary Least-Squares

  33. Geometric interpretation • b is a vector in Rn Ordinary Least-Squares

  34. Geometric interpretation • b is a vector in Rn • The columns of A define a vector space range(A) Ordinary Least-Squares

  35. Geometric interpretation • b is a vector in Rn • The columns of A define a vector space range(A) • Ax is an arbitrary vector in range(A) Ordinary Least-Squares

  36. Geometric interpretation • b is a vector in Rn • The columns of A define a vector space range(A) • Ax is an arbitrary vector in range(A) Ordinary Least-Squares

  37. Geometric interpretation • is the orthogonal projection of b onto range(A) Ordinary Least-Squares

  38. The normal equation: Ordinary Least-Squares

  39. The normal equation: • Existence: has always a solution Ordinary Least-Squares

  40. The normal equation: • Existence: has always a solution • Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

  41. The normal equation: • Existence: has always a solution • Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

  42. Under-constrained problem Ordinary Least-Squares

  43. Under-constrained problem Ordinary Least-Squares

  44. Under-constrained problem Ordinary Least-Squares

  45. Under-constrained problem • Poorly selected data • One or more of the parameters are redundant Ordinary Least-Squares

  46. Under-constrained problem • Poorly selected data • One or more of the parameters are redundant Add constraints Ordinary Least-Squares

  47. How good is the least-squares criteria? • Optimality: the Gauss-Markov theorem Ordinary Least-Squares

  48. How good is the least-squares criteria? • Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: Ordinary Least-Squares

  49. How good is the least-squares criteria? • Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Ordinary Least-Squares

  50. How good is the least-squares criteria? • Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Then is the best unbiased linear estimator Ordinary Least-Squares

More Related