Effects of unequal indicator intercepts on manifest composite differences

1. Effects of unequal indicator intercepts on manifest composite differences Holger Steinmetz and Peter Schmidt University of Giessen / Germany

2. Introduction • Importance of analyses of mean differences For instance: • gender differences on wellbeing, self-esteem, abilities, behavior • differences between leaders and non-leaders on intelligence and personality traits • differences between cultural populations on psychological competencies, values, wellbeing • Usual procedure: t-test or ANOVA with manifest composite scores • Latent variables vs. manifest variables • Manifest mean = indicator intercept + factor loading * latent mean • → Will unequal intercepts lead to wrong conclusions regarding composite differences?

3. B1 Y X B0 Y X Intercepts and latent means

4. xi l1 d1 x1 l2 d2 x2 x li l3 x3 d3 l4 ti x4 d4 x Intercepts and latent means

5. xi l1 d1 x1 l2 d2 x2 x li l3 x3 d3 l4 ti x4 d4 x Intercepts and latent means

6. x1 x2 x M(xi) x3 x4 k Intercepts and latent means xi l1 d1 l2 d2 li l3 d3 l4 ti d4 x

7. xi x1 li x2 x M(xi) x3 ti x4 k Intercepts and latent means l1 d1 l2 d2 l3 d3 l4 d4 x

8. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi M(xi) M(xi) M(xi) x k

9. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi M(xi) M(xi) M(xi) x k

10. x1 x1 x2 x2 x x x3 x3 x4 x4 Group differences in intercepts and factor loadings Group A Group B xi M(xi) M(xi) M(xi) x k

11. Meaning of (unequal) intercepts • Associated terms used in the literature • Item bias • Differential item functioning • Measurement/factorial invariance ("strong factorial invariance", "scalar invariance") • Meaning • Response style (acquiescence, leniency, severity) • Response sets (e.g., social desirability) • Connotations of items • Item specific difficulty

12. The study • Partial invariance • Research question: Is partial invariance enough for composite mean difference testing? • Pseudo-differences • Compensation effects • Procedure (Mplus): • Step 1: Specification of two-group population models with latent mean and intercept differences; 1000 replications, raw data saved • Step 2: Creation of a composite score • Step 3: Analysis of composite differences • Step 4: Aggregation (-> sampling distribution)

13. The study • Design (population model): • Two groups • One latent variable • 4 vs. 6 indicators • All intercepts equal vs. one vs. two intercepts unequal in varying directions (+.30 vs. -.30) • Latent mean difference: 0 vs. .30 • Loadings kept equal with l‘s = .80; latent variance = 1 • N = 2 x 100 vs. 2 x 300 • Latent models as comparison standard for each condition • Dependent variables • Average composite mean difference • Percent of significant composite differences („% sig“)

14. 1 Avg. composite difference 0.9 %sig 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 6 Indicators 6 Indicators 4 Indicators 4 Indicators N = 2 x 100 N = 2 x 300 Full scalar invariance(Latent mean difference = .30)

15. 0.60 0.50 Avg. composite difference %sig 0.40 0.30 0.20 0.10 0.00 6 Ind. 4 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 2 Intercepts unequal (.30) 1 Intercept unequal (.30) N = 2 x 300 N = 2 x 100 N = 2 x 300 N = 2 x 100 Pseudo-Differences(Latent mean difference = 0; unequal intercept(s)

16. Pseudo-Differences(Latent mean difference = 0; unequal intercept(s) 0.60 0.50 Avg. composite difference %sig 0.40 0.30 0.20 0.10 0.00 6 Ind. 4 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 2 Intercepts unequal (.30) 1 Intercept unequal (.30) N = 2 x 300 N = 2 x 100 N = 2 x 300 N = 2 x 100

17. 6 Ind. 4 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 2 Intercepts unequal (-.30) 1 Intercept unequal (-.30) N = 2 x 300 N = 2 x 100 N = 2 x 300 N = 2 x 100 Compensation effects(Latent mean difference = .30; negative intercept difference)

18. Compensation effects(Latent mean difference = .30; negative intercept difference) 6 Ind. 4 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 4 Ind. 6 Ind. 2 Intercepts unequal (-.30) 1 Intercept unequal (-.30) N = 2 x 300 N = 2 x 100 N = 2 x 300 N = 2 x 100

19. Summary • Full latent variable models have more power than composite analyses • Pseudo-differences • Even one unequal intercept increases the risk to find spurious composite differences • High sample size increases risk • Number of indicators reduces the risk – but not substantially • Componensation effects • Even one unequal intercept reduces the size of the composite difference to 50% • In small samples little chance to find a significant composite difference (power = .25 - .40) • Two unequal intercepts drastically reduce the composite difference: The power in the „best“ condition (2x300, 6 Ind.) is only .50

20. Conclusons • Most comparisons of means rely on traditional composite difference analysis • These methods make assumptions that are unrealistic (i.e., full invariance of intercepts) • Even minor violations of these assumptions increase the risk of drawing wrong conclusions • Advantages of SEM: • Assumptions can be tested • Partial invariance implies no danger • Greater power even in small samples

21. Thank you very much!