Use of SEM programs to precisely measure scale reliability

1. IMPS2001, July 15-19,2001Osaka, Japan Use of SEM programs to precisely measure scale reliability Yutaka Kano and Yukari Azuma Osaka University

2. Reliability measure for • Reliability with possibly correlated errors • α coefficient

3. An example α 0.69 0.74 0.78ρ’ 0.69 0.64 0.60

4. Coefficient alpha can be distorted seriously by error correlations e.g. Green-Hershberger (2000), Raykov (2001) In the case, ρ’ has to be used to correctly figure out the reliability From the example

5. Problem • How can one identify error correlations? • The factor model allowing for (fully) correlated errors is not identifiable, because it contains too many parameters • A trivial solution would be It does not work because

6. LM approach • Start from the factor model with no error correlation • Perform the LM test for releasing a zero covariance between errors using a SEM program • EQS can perform it most easily and most accurately

7. Real data analysis • A questionnaire on perception on physical exercise • n=653, p=15, one-factor model • The data were collected by Dr. Oka (Waseda U.)

8. Result_1 • Best fitted model, with 7 correlated errors • χ2=250.375(df=83) (n=653) • GFI=0.950, CFI=0.952, RMSEA=0.056

9. Result_2 • Estimates of reliability • α = 0.90 • ρ’ = 0.90 by • ρ’ = 0.87 by LM test • Note that

10. Search for variables • Even though one factor model is fitted well, inclusion of a variable with small true variance can reduce reliability • There is no convenient way to select variables for the composite scale to have maximum reliability

11. Mathematically… • It is complicated • It will become more complicated if error correlations are allowed

12. New program • A new program is being developed which • gives a list of reliability estimates for each factor; • gives a list of predicted reliability estimates when one variable is removed • Error correlations are allowed

13. Flowchart Well fit? No DATA Free some error covariances to get good fit Factor analysis Yes Decide composite scale items Print reliability End

14. ρ’ = 0.87 with 15 variables Example, continued

15. Scale developer

16. If V13 is removed, then…

17. Results • For one-factor model with uncorrelated errors, the variable with the smallest factor loading is least favorable. • If there is a variable whose deletion improves reliability, then this is the variable. • For one-factor model with correlated errors, the variable with the smallest factor loading is not always least favorable. • While deletion of the variable does not improve reliability, there may be other variables to be deleted to improve reliability. • The example here is the case.

18. Summary • Correlated errors invalidate the coefficient alpha and traditional one-factor based reliability. • LM test is useful to find error correlations. • Magnitude of factor loadings does not necessarily provide accurate information on indicator selection when correlated errors exist. • The forthcoming Web-based program will help reliability analysis.