Simple and multiple regression analysis in matrix form

1. Simple and multiple regression analysis in matrix form • Leastsquare • Betaestimation • Simple linear regression • Multiple regression with two predictors • Multiple regression with three predictors • Sum of square • R2 • Test on bparameters • Covariance matrix of the b • Standard error of the b

2. Simple and multiple regression analysis in matrix form • Tests on individual predictors • Variance of individual predictors • Correlation between predictors • Standardized matrices • Correlation matrices • Sum of squares in Z • R2 in Z • R2between independent variables • Standard error of b in Z

3. Least square Starting from the general: The method of least squares estimate of the beta parameter minimizing the sum of squares due to error. In fact, if:

4. Least square You can estimate:

5. Simple linear regression

6. Simple linear regression

7. Simple linear regression

8. Simple linear regression intercepts slope

9. Multiple regression • Similar to the simple • A single dependent variable (Y) • Two or more independent variables (X) • Multiple correlation (rather than simple) • Estimation by least squares

10. Multiple regression Simple linear regression (var.: 1 dep., 1 indep.) Independent variables intercepts slope error Multiple linear regression (Var.:1 dep., 2 indep.)

11. Multiple regression matrix form

12. Multiple regression matrix form

13. Multiple regression matrix form

14. Multiple regression matrix form X’X inversa

15. Multiple regression matrix form

17. Multiple regression with three predictors In matrix notation is briefly expressed :

18. Multiple regression with three predictors

19. Matrix form

20. Matrix form

21. Matrix form

22. Matrix form

23. General scheme

24. General scheme

25. Sum squares The least squares method allows to check the following equality:

26. Sum squares Since in general: it's possible to derive that the sum of the squares of the distances of y from its average can be decomposed into the sum of squares due to regression and the sum of squares due to error, according to:

27. Sum squares It should be noted the equivalence of :

28. Sum squares

29. Sum squares

30. Sum squares In summary :

31. R2

32. Adjusted R2YY’ Because the coefficient of determination depends on both the number of observations (n) that the number of independent variables (k) it is convenient to correct by the degrees of freedom. Adjusted R2YY’ In our example :

33. Test on b parameters • Once a regression model has been constructed, it may be important to confirm the goodness of fit(R-squared )of the model and the statistical significance of the estimated parameters. Statistical significance can be checked by an F-testof the overall fit, followed by t-tests of individual parameters

34. Test on b parameters • You can test the hypothesis of differences with 0 of the parameters bi taken together :

35. Test on b parameters k= Number of columns of the matrix X excluding X0 n= Number of observations in y

36. Test on b parameters k= Number of columns of the matrix X excluding X0 n= Number of observations in y

37. Covariance matrix of the b We denote: An estimate of the covariance matrix of the beta values result by:

38. Covariance matrix of the b Where the diagonal elements are an estimate of the variance of the single bi

39. Standard error of the b The standard error of the parameters can be calculated with the following formula: whereciiis the diagonal element inside the matrix(X’X)-1 corresponding to the parameter bi .

40. Standard error of the b Nota: quando il valore di cii è elevato il valore di sebi cresce, indicando che la variabile Xi ha un alto coefficiente di correlazione multipla con le altre variabili X.

41. Standard error of the b The standard error of the i can also be calculated in the following way: where the increase in R2i led to a decreases of the denominator of the ratio and, consequently, increases the value of the standard error of the parameterbi.

42. Tests on individual predictors • With the standard error of measurement associated with each biyou can make a t-test to verify:

43. Tests on individual predictors With the standard error of measurement associated with each bi is also possible to estimate the confidence interval for each parameter:

44. Tests on individual predictors In order to conduct a statistical test on the regression coefficients is necessary: • Calculate the SSreg for the model containing all the independent variables. • Calculate the SSreg for the model excluding the variable for which you want to test the significance (SS-i). • Perform an F-test with the numerator equal to the difference SSreg-SSi weighted for the difference between the degrees of freedom of the two models, and with denominator SSREs / (nk-1).

45. Tests on individual predictors To test, for example, only the weight of the first predictor compared to the total model, it is necessary to calculate a new matrix bi from the matrix Xi which was taken off the column belonging to the first predictor. From this follows immediately the calculation of SSi.

46. Tests on individual predictors

47. Tests on individual predictors Similarly we have: Same procedure is followed to test any subset of predictors.

48. Tests on individual predictors It is interesting to note that this test on a single predictor is equivalent to the t-test b1 = 0. When the numerator there is only one degree of freedom, that is in fact the equivalence:

49. Summary table On this occasion, none of the estimated parameters obtained statistical significance on the hypothesis bi 0

50. Variance of individual predictors Xi Using the matrix X'X we can calculate the variance of each variable Xi .