1 / 23

Topic 11: Matrix Approach to Linear Regression

Topic 11: Matrix Approach to Linear Regression. Outline. Linear Regression in Matrix Form. The Model in Scalar Form. Y i = β 0 + β 1 X i + e i The e i are independent Normally distributed random variables with mean 0 and variance σ 2 Consider writing the observations:

zelenia-melton
Download Presentation

Topic 11: Matrix Approach to Linear Regression

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Topic 11: Matrix Approach to Linear Regression

  2. Outline • Linear Regression in Matrix Form

  3. The Model in Scalar Form • Yi = β0 + β1Xi + ei • Theeiare independent Normally distributed random variables with mean 0 and variance σ2 • Consider writing the observations: Y1= β0 + β1X1 + e1 Y2= β0 + β1X2 + e2 : Yn= β0 + β1Xn + en

  4. The Model in Matrix Form

  5. The Model in Matrix Form II

  6. The Design Matrix

  7. Vector of Parameters

  8. Vector of error terms

  9. Vector of responses

  10. Simple Linear Regression in Matrix Form

  11. Variance-Covariance Matrix Main diagonal values are the variances and off-diagonal values are the covariances.

  12. Covariance Matrix of e Independent errors means that the covariance of any two residuals is zero. Common variance implies the main diagonal values are equal.

  13. Covariance Matrix of Y

  14. Distributional Assumptions in Matrix Form • e ~ N(0, σ2I) • I is an n x nidentity matrix • Ones in the diagonal elements specify that the variance of each ei is 1 times σ2 • Zeros in the off-diagonal elements specify that the covariance between different ei is zero • This implies that the correlations are zero

  15. Least Squares • We want to minimize (Y-Xβ)(Y-Xβ) • We take the derivative with respect to the (vector) β • This is like a quadratic function • Recall the function we minimized using the scalar form

  16. Least Squares • The derivative is 2 times the derivative of (Y-Xβ) with respect to β • In other words, –2X(Y-Xβ) • We set this equal to 0 (a vector of zeros) • So, –2X(Y-Xβ) = 0 • Or, XY = XXβ (the normal equations)

  17. Normal Equations • XY = (XX)β • Solving for β gives the least squares solution b = (b0, b1) • b = (XX)–1(XY) • See KNNL p 199 for details • This same matrix approach works for multiple regression!!!!!!

  18. Fitted Values

  19. Hat Matrix We’ll use this matrix when assessing diagnostics in multiple regression

  20. Estimated Covariance Matrix of b • This matrix, b, is a linear combination of the elements of Y • These estimates are Normal if Y is Normal • These estimates will be approximately Normal in general

  21. A Useful MultivariateTheorem • U ~ N(μ, Σ), a multivariate Normal vector • V = c + DU, a linear transformation of U c is a vector and D is a matrix • Then V ~ N(c+Dμ, DΣD)

  22. Application to b • b = (XX)–1(XY) = ((XX)–1X)Y • Since Y ~ N(Xβ, σ2I) this means the vector b is Normally distributed with mean (XX)–1XXβ = β and covariance σ2 ((XX)–1X)I((XX)–1X) = σ2 (XX)–1

  23. Background Reading • We will use this framework to do multiple regression where we have more than one explanatory variable • Another explanatory variable is comparable to adding another column in the design matrix • See Chapter 6

More Related