Multiple Linear Regression

12 Multiple Linear Regression CHAPTER OUTLINE 12-1 Multiple Linear Regression Model 12-1.1 Introduction 12-1.2 Least squares estimation of the parameters 12-1.3 Matrix approach to multiple linear regression 12-1.4 Properties of the least squares estimators 12-2 Hypothesis Tests in Multiple Linear Regression 12-2.1 Test for significance of regression 12-2.2 Tests on individual regression coefficients & subsets of coefficients 12-3 Confidence Intervals in Multiple Linear Regression 12-4.1 Use of t-tests 12-3.2 Confidence interval on the mean response 12-4 Prediction of New Observations 12-5 Model Adequacy Checking 12-5.1 Residual analysis 12-5.2 Influential observations 12-6 Aspects of Multiple Regression Modeling 12-6.1 Polynomial regression models 12-6.2 Categorical regressors & indicator variables 12-6.3 Selection of variables & model building 12-6.4 Multicollinearity

12-1: Multiple Linear Regression Models 12-1.1 Introduction • Many applications of regression analysis involve situations in which there are more than one regressor variable. • A regression model that contains more than one regressor variable is called a multiple regression model.

12-1: Multiple Linear Regression Models 12-1.1 Introduction • For example, suppose that the effective life of a cutting tool depends on the cutting speed and the tool angle. A possible multiple regression model could be where Y – tool life x1 – cutting speed x2 – tool angle β1 and β2 partial regression coefficients • β1 measures the expected change in Y per unit change in x1 when x2 is held constant, • β2 measures the expected change in Y per unit change in x2 when x1 is held constant.

12-1: Multiple Linear Regression Models 12-1.1 Introduction Figure 12-1(a)The regression plane for the model E(Y) = 50 + 10x1 + 7x2. (b) The contour plot

12-1: Multiple Linear Regression Models 12-1.1 Introduction In general, any regression model that is linear in parameters (the β’s) isa linear regression model, regardless of the shape of the surface that it generates.

12-1: Multiple Linear Regression Models 12-1.1 Introduction Figure 12-2 (a) Three-dimensional plot of the regression model E(Y) = 50 + 10x1 + 7x2 + 5x1x2. (b) The contour plot

12-1: Multiple Linear Regression Models 12-1.1 Introduction Figure 12-3 (a) Three-dimensional plot of the regression model E(Y) = 800 + 10x1 + 7x2 – 8.5x12 – 5x22 + 4x1x2. (b) The contour plot

12-1: Multiple Linear Regression Models 12-1.2 Least Squares Estimation of the Parameters

12-1: Multiple Linear Regression Models 12-1.2 Least Squares Estimation of the Parameters • The least squares function is given by • The least squares estimates must satisfy

12-1: Multiple Linear Regression Models 12-1.2 Least Squares Estimation of the Parameters • Simplifying Equation 12-9, we obtain the least squares normal equations • The solution to the normal Equations are the least squares estimators of the regression coefficients.

12-1: Multiple Linear Regression Models Example 12-1

12-1: Multiple Linear Regression Models • For example, the plot indicates that there is a strong linear relationship between strength and wire length.

12-1: Multiple Linear Regression Models 12-1.3 Matrix Approach to Multiple Linear Regression Suppose the model relating the regressors to the response is In matrix notation this model can be written as

12-1: Multiple Linear Regression Models 12-1.3 Matrix Approach to Multiple Linear Regression where

12-1: Multiple Linear Regression Models 12-1.3 Matrix Approach to Multiple Linear Regression We wish to find the vector of least squares estimators that minimizes: The resulting least squares estimate is

12-1: Multiple Linear Regression Models 12-1.3 Matrix Approach to Multiple Linear Regression

Example 12-2

12-1: Multiple Linear Regression Models Estimating 2 An unbiased estimator of 2 is

12-1: Multiple Linear Regression Models 12-1.4 Properties of the Least Squares Estimators Unbiased estimators: Covariance Matrix:

12-1: Multiple Linear Regression Models 12-1.4 Properties of the Least Squares Estimators Individual variances and covariances: In general, In general, the covariance matrix of β is a ( p × p) symmetric matrix whose jjth element is the variance of β j and whose i, jth element is the covariance between βi and β j

12-2: Hypothesis Tests in Multiple Linear Regression 12-2.1 Test for Significance of Regression The test for significance of regression is a test to determine if there is a linear relationship between the response y and any of the regressor variables x1, x2, . . . , xk . This procedure is often thought of as an overall or global test of model adequacy. The appropriate hypotheses are The test statistic is RejectH0 :β1 =β2 =···=βk =0ifF0 >Fα,ν1,ν2. ν1 = k and ν2 = n − p.

12-2: Hypothesis Tests in Multiple Linear Regression 12-2.1 Test for Significance of Regression

12-2: Hypothesis Tests in Multiple Linear Regression 12-2.1 Test for Significance of Regression Decomposition of total sum of squares SST = SSReg + SSRes

12-2: Hypothesis Tests in Multiple Linear Regression Example 12-3

12-2: Hypothesis Tests in Multiple Linear Regression R2 and Adjusted R2 • Two other ways to assess the overall adequacy of the model are R2and adjusted R2, denoted R2Adj . • In general, R2never decreases when a regressor is added to the model, regardless of the value of the contribution of that variable. Therefore, it is difficult to judge whether an increase in R2is really telling us anything important. • R2Adj will only increase on adding a variable to the model if the addition of the variable reduces the residual mean square.

12-2: Hypothesis Tests in Multiple Linear Regression R2 and Adjusted R2 The coefficient of multiple determination • For the wire bond pull strength data, we find that R2 = SSR/SST = 5990.7712/6105.9447 = 0.9811. • Thus, the model accounts for about 98% of the variability in the pull strength response.

12-2: Hypothesis Tests in Multiple Linear Regression R2 and Adjusted R2 The adjusted R2 is • The adjusted R2 statistic penalizes the analyst for adding terms to the model. • It can help guard against overfitting (including regressors that are not really useful)

12-2: Hypothesis Tests in Multiple Linear Regression 12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients The hypotheses for testing the significance of any individual regression coefficient:

12-2: Hypothesis Tests in Multiple Linear Regression 12-2.2 Tests on Individual Regression Coefficients and Subsets of Coefficients The test statistic is • Reject H0 if |t0| > t/2,n-p. • This is called a partial or marginal test • If H0 : βj = 0 is not rejected, then this indicates that the regressor xj can be deleted from the model.

12-2: Hypothesis Tests in Multiple Linear Regression The general regression significance test or the extra sum of squares method: We wish to test the hypotheses:

12-2: Hypothesis Tests in Multiple Linear Regression A general form of the model can be written: where X1 represents the columns of X associated with 1 and X2 represents the columns of X associated with 2

12-2: Hypothesis Tests in Multiple Linear Regression For the full model: If H0 is true, the reduced model is

12-2: Hypothesis Tests in Multiple Linear Regression The test statistic is: Reject H0 if f0 > f,r,n-p The test in Equation (12-32) is often referred to as a partial F-test

Multiple Linear Regression