Relative Importance of Predictors with Regression Models

1. Relative Importance of Predictors with Regression Models James M. LeBreton Purdue University

2. Multiple Regression: Prediction • Specify a regression equation within a sample to predict scores on a single criterion variable (DV) from multiple predictor variables (IVs). • Estimate the regression coefficients. • Apply these coefficients in similar samples where data are available on the predictor variables but not on the criterion variable. • The R2 statistic reveals the strength of association between the criterion and the weighted linear combination of the predictors.

3. Multiple Regression: Prediction • “The extent to which the criterion can be predicted by the predictor variables … is of much greater interest than is the relative magnitude of the regression coefficients” (pp. 238-239; Johnson & LeBreton, 2004). • “When our research application is purely predictive, in a sense interpretation may be irrelevant. We may very much desire an accurate prediction, but we may not care why our predictive rule works” (p. 232; Courville & Thompson, 2001).

4. Multiple Regression: Explanation • Multiple regression may also be used for theory-testing or explanation purposes. • In such instances prediction is important, but so too is an understanding of why the regression equation “works.” • Thus, interpretation of the regression model becomes the primary focus of researchers.

5. Multiple Regression: Explanation • “In theory testing or explanation applications, interpretation is very relevant. We want to know how useful the variables are.” (p. 233; Courville & Thompson, 2001). • “In this case, we are very interested in the extent to which each variable contributes to the prediction of the criterion …interpretation is the primary concern, such that substantive conclusions can be drawn regarding one predictor with respect to another” (p. 239, Johnson & LeBreton, 2004).

6. Multiple Regression: Explanation & Relative Importance • Conducting a relative importance analysis may help researchers who are using regression for theory-testing or explanation purposes (Johnson & LeBreton, 2004; LeBreton, Hargis, Griepentrog, Oswald, & Ployhart, 2007). • Relative importance refers to the proportionate contribution each predictor makes to R2 considering both its individual effect and its effect when combined with the other variables in a regression equation (Budescu, 1993; Johnson & LeBreton, 2004). • Thus, a relative importance analysis provides information about the relative contributions predictors make to the overall explanatory power of the regression model.

7. Relative Importance Y X1 X2

8. Relative Importance Y X5 X1 X2 X3 X4

9. Relative Importance R2 = .60 % attributable to X5? % attributable to X1? % attributable to X2? % attributable to X3? % attributable to X4?

10. Who should care about relative importance? • Researchers who utilize multiple linear regression for theory-testing or explanation purposes. • Researchers who want to understand which variables are contributing most to a regression model. • Relative importance tells you what variable or variables are “driving” the R2. • The information obtained from a relative importance analysis supplements the information obtained from a traditional multiple linear regression analysis.

11. Sample Research Questions • Performance Evaluation: • Which dimensions of contextual performance are most important for predicting promotion and turnover? • Work-Family Conflict: • What are the most important situational and dispositional predictors of FIW conflict and WIF conflict? • Motivation: • What is the relative contribution of conscientiousness, goal orientation, and cognitive ability to the prediction of college GPA?

12. More general research questions • Health Behavior: • How do impulsivity, message framing, and attitudes about smoking contribute to the prediction of smoking cessation? • Adolescent Adjustment: • What is the relative importance of negative affect (e.g., hostility, depression, anxiety, guilt), social skills, and parenting style in predicting adolescent autonomy?

13. Regression vs. Relative Importance • Traditional Regression Analysis • Estimate structural parameters that are used to describe data or construct prediction equations to forecast future criterion performance. • Relative Importance Analysis • Explore the relative contribution that regressors make to the model R2. • Permit comparisons among the regressors in terms of which are most important.

14. Measures of Relative Importance • Johnson & LeBreton (2004) reviewed various measures of relative importance and their use in organizational research. • Bivariate correlations • Beta weights • Product measure weights • General dominance weights • Relative weights

15. (Squared) Zero-Order Correlations • Limited because, by definition, they only inform us about the bivariate relationship. • They ignore all information about the other predictors included in a regression model. • Squared correlations only sum to the model R2 when the predictors are orthogonal.

16. (Squared) Beta Weights • Most commonly used indices of importance. • Sum to the model R2 only when the predictors are orthogonal. • Possible for variables with large correlations to have near-zero or negative estimates of importance.

17. Standardized Regression Coefficients • What do betas and squared betas really tell you? • Betas really inform us about the change in ZY associated with a unit increase in a particular ZX holding all other ZXs constant. • Squared betas only inform us about relative importance when the predictors are orthogonal.

18. Problems with Betas: Matrix 1 • b21 and b22 =.0386 while b23 and b24 = .0015. • Conclusion: • X1 and X2 are roughly 25 times more “important” than X3 and X4.

19. Problems with Betas: Matrix 2 • b21 = b22 = .0672; b23=.0001 and b24 =.0043. • Conclusion: • X1 and X2 are roughly 16 times more “important” than X4 and 672 times more important than X3..

20. Product Measure • Product of the zero-order correlation and standardized regression coefficient. • Less common in organizational research. • Advantage • Sums to the model R2. • Disadvantages • Shares the limitations of both measures of which it is composed. • Negative values • Near zero values

21. Problems with Product: Matrix 2 Revisited • Product1=.1037; Product2=.1037; Product3=.0026; Product4=-.0197 • Conclusion: • X4 actually has negative importance…remember these sum to the R2 and may be interpreted as relative effect sizes. Thus, X4 actually explains negative variance according to the product measure..

22. Summary • A variety of techniques exist for estimating relative importance. • Most of the early techniques yield identical estimates when the predictors are uncorrelated. • However, problems emerge for these techniques when there are even moderate correlations among the predictors.

23. Estimating Importance with Correlated Predictors • Problems associated with estimating relative importance when one has correlated predictors were identified over 40 years ago (Darlington,1968). • However, the last 15 years substantial progress has been made towards accurately assessing relative importance among correlated predictors. • General dominance weights (Budescu, 1993). • Relative weights (Johnson, 2000).

24. General Dominance Weights (Budescu, 1993) • General dominance weights are defined as the average contribution that a variable makes to the R2 across all possible subset regressions. • Average squared semi-partial correlation.

25. Sample Dominance Analysis -> Dependent variable = Job Satisfaction (Rsq=.1548) Squared Semipartial Correlations ______(DV = JOBSAT) RSq EXTRA STABLE AGREE__ None 0.0478 0.1296 0.0308 EXTRA 0.0478 0.1058 0.0273 STABLE 0.1296 0.0241 0.0021 AGREE 0.0308 0.0443 0.1009 EXTRA STABLE 0.1536 0.0012 EXTRA AGREE 0.0751 0.0797 ________STABLE AGREE 0.1317 0.0231 _____________________ General Dominance Weights: 0.0350 0.1042 0.0156 Rescaled Weights: 0.2263 0.6731 0.1006

26. Beyond the Weights: Patterns of Dominance • Azen & Budescu (2003) defined patterns of dominance. • Completedominance • For example, four predictors X1, X2,X3, and X4. • X1 is said to “completely dominate” X2if X1 has a larger squared semi-partial correlation compared to X2across all possible models that include both of those variables. • This is denoted, X1> D > X2. • Conditional dominance • Alternatively, X1 might dominate X2 for some models (e.g., when X3 is included) but not others (e.g., when X3 and X4 are included). • General dominance • General dominance simply focuses on the importance weights.

27. Beyond the Weights: Patterns of Dominance • Thus, we can examine both the magnitude and pattern ofavariable’s relative importance. • As a result, psychologists can use dominance analysis to identify variables that tend to “outperform” other variables across the various regression models.

28. Sample Dominance Analysis -> Dependent variable = Job Satisfaction (Rsq=.1548) Squared Semipartial Correlations ______(DV = JOBSAT) RSq EXTRA STABLE AGREE__ None 0.0478 0.1296 0.0308 EXTRA 0.0478 0.1058 0.0273 STABLE 0.1296 0.0241 0.0021 AGREE 0.0308 0.0443 0.1009 EXTRA STABLE 0.1536 0.0012 EXTRA AGREE 0.0751 0.0797 ________STABLE AGREE 0.1317 0.0231 _____________________ General Dominance Weights: 0.0350 0.1042 0.0156 Rescaled Weights: 0.2263 0.6731 0.1006 STABLE >D> EXTRA >D> AGREE

29. Summary: Dominance Analysis • Advantages • Specifically developed for use with correlated predictors • Estimates of relative importance sum to R2 • Estimates are intuitively meaningful • Can explore patterns of dominance • Disadvantages • Computationally intensive: dominance analysis containing 20 predictors involves computing 1,048,575 regressions

30. # of Subset Regression Models Needed for a Dominance Analysis

31. Relative Weights(Johnson, 2000) • Relative weights (or Epsilon Weights) are calculated by creating a new set of uncorrelated predictors (ZK) that are maximally related to the original set of correlated predictors (XJ). • Both Zs and Xs are used to estimate importance. • Y is regressed on Zs • Zs are regressed on Xs • Relative weights are basically the product of squared standardized regression weights obtained in steps 1 and 2

32. Relative Weight Analysis X1 Z1 11 b1 12 13 21 b2 X2 Y Z2 22 23 b3 31 Z3 X3 32 33

33. Relative Weight Analysis: Matrix Approach • A decomposition of the matrix of predictor intercorrelations RXX exists: • Where contains the eigenvectors obtained from RXX and is a diagonal matrix containing the eigenvalues obtained from RXX. • Johnson (2000) obtained the square factorization of RXX : • These lambda weights yield the correlations between the orthogonal predictors and the original correlated predictors. • We continue by obtaining the regression coefficients linking the correlations between the orthogonal variables and the criterion: • Finally, the elements of and are squared and multiplied by one another to obtain relative weights:

34. Summary: Relative Weight Analysis • Advantages • Specifically developed for use with correlated predictors • Estimates of relative importance sum to R2 • Estimates are intuitively meaningful (i.e., relative effect sizes) • Easy to calculate • Disadvantage • The primary limitation of the relative weights is that the importance analysis is typically conducted on the full model containing all predictors.

35. Dominance Analysis vs. Relative Weight Analysis • Dominance analysis permits researchers to easily conduct conditional or constrained importance analyses (Azen & Budescu, 2003). • Such analyses specify, a priori, a variable (or group of variables) that will be included as part of a baseline model. • e.g., cognitive ability will be used to predict performance • Relative importance of the remaining variables is then tested. • e.g., test the relative importance of the Big 5 personality traits, biodata, and emotional labor for predicting performance, but only when cognitive ability is included in the model

36. How do the methods compare empirically? • Dominance and epsilon are based on very different statistical rationales. • Dominance analysis deals with collinearity using an all subset regression approach. • Relative weight analysis deals with collinearity using a variable transformation approach. • However, both are designed to deal with collinearity and both furnish estimates that sum to the R2. • So, do these importance estimate converge or diverge? How do they relate to betas?

37. Comparisons • Johnson (2000) • Predicted peer-ratings of overall job performance using self-ratings of various traits. • e.g., dependability, friendliness • Results revealed virtually identical results between general dominance weights and relative weights (N=631). • However, these newer techniques deviated substantially from the results obtained using squared correlations, squared betas, and the product measure.

38. Comparisons • LeBreton, Ployhart & Ladd (2004) • Monte Carlo comparison of squared correlations, squared betas, the product measure, relative weights, and general dominance weights. • Compared how these techniques rank-ordered the predictors. • Manipulated various features of the data • Criterion-Related Validity • Collinearity • Number of total predictors in regression • Found that relative weights and general dominance weights yielded virtually identical results and betas yielded the most discrepant.

39. Convergence Between Dominance and Relative Weights Average Kendall’s Tau Correlation Between Dominance Ranking of Predictors And Beta Ranking of Predictors Average Kendall’s Tau Correlation Between Dominance Ranking of Predictors And Epsilon Ranking of Predictors

40. Convergence Between Dominance and Relative Weights Average Kendall’s Tau Correlation Between Dominance Ranking of Predictors And Beta Ranking of Predictors Average Kendall’s Tau Correlation Between Dominance Ranking of Predictors And Epsilon Ranking of Predictors

41. Recommendations for Organizational Researchers • Use dominance analysis or relative weight analysis - they “make sense.” • Sum to the model R2 • Appropriate for use with correlated predictors • Intuitively appealing metric • Yield highly convergent results • Standardized coefficients do not make “make sense.” • Rarely sum to the model R2 • Inappropriate for use with correlated predictors • Yield highly divergent results compared to dominance and epsilon

42. Assessing Importance for Multidimensional Criteria • Organizational research often involves models involving the measurement and testing of multidimensional constructs. • Job Satisfaction • Supervisor vs. Pay vs. Coworkers • Organizational Commitment • Affective vs. Calculative vs. Normative • Job Performance • Task vs. Citizenship vs. Counterproductive

43. Assessing Importance for Multidimensional Criteria • Azen & Budescu (2006) • Extended Budescu’s (1993) general dominance weights to multivariate multiple regression designs. • After identifying a multivariate analog of R2, their procedure is identical to univariate dominance. • Basically, they estimate the usefulness of each predictor across all subset regression models using one of two multivariate analogs of R2. • They demonstrated that failure to take into account the correlations among the criteria can be problematic.

44. Assessing Importance for Multidimensional Criteria • LeBreton & Tonidandel (in press) • Extended Johnson’s (2000) relative weights to multivariate multiple regression designs. • Based on the same logic, but requires a few extra steps because now the collinearity among the criteria need to be considered.

45. Assessing Importance for Multidimensional Criteria

46. Assessing Importance for Multidimensional Criteria • LeBreton & Tonidandel (in press) explored the convergence between multivariate relative weights and multivariate general dominance weights. • Consistent with their univariate analogs…the multivariate estimates converged almost perfectly with one another. • Consistent with Azen & Budescu (2006), univariate estimates were not necessarily consistent with multivariate estimates.

47. Significance of Relative Importance Estimates • Practical Significance • Relative weights and general dominance weights may be interpreted as estimates of relative effect size.

48. Significance of Relative Importance Estimates • Statistical Significance • Johnson (2004) – recommended bootstrapping standard errors to conduct significance tests and/or calculate confidence intervals.

49. Significance of Relative Importance Estimates • Summary • Development of practical heuristics related to effect size estimates may be useful for ascertaining “practical” significance. • Techniques for estimating the statistical significance of relative weights and general dominance weights are still being developed.

50. Evaluating New Variables in Organizational Research: An Integrative Approach • Traditional Approach: • Does a new variable demonstrate incremental validity above and beyond the existing variables in the regression? • Pros: • Necessary to demonstrate our new variable is not redundant with existing variables. • Cons: • Most new variables are not included in a regression to account for small portions of “unique” variance. • Instead, they are designed to be highly correlated with the criterion.