1 / 58

The Structural Equation Modeling approach

The Structural Equation Modeling approach. The effect of survey characteristics on random and systematic errors in surveys by Willem Saris. Designing questions.

palmquist
Download Presentation

The Structural Equation Modeling approach

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Structural Equation Modeling approach The effect of survey characteristics on random and systematic errors in surveys by Willem Saris

  2. Designing questions • Questionnaire design consists of a series of decisions: topic, concept, form of the question, response categories etc. (Andrews 1984). • If we know the consequences of our decisions • questionnaire design can be done in a scientific way • This idea is the basic motivation of our long term dedication to our research on the effects of survey characteristics on data quality college titel en nummer

  3. An Illustration • Three ESS questions about satisfaction: • On the whole how satisfied are you with the present state of the economy in Britain ? • Now think about the national government. How satisfied are you with the way it is doing its job ? • And on the whole, how satisfied are you with the way democracy works in Britain ? college titel en nummer

  4. Three alternative response scales • A first method • 1 Very satisfied , 2 fairly satisfied, 3 fairly dissatisfied or 4 very dissatisfied • A second method • very very • dissatisfied Satisfied • 0 1 2 3 4 5 6 7 8 9 10 • A third method: • 1 not at all satisfied 2 satisfied 3 rather satisfied 4 very satisfied college titel en nummer

  5. Does it matter what we choose ? • The combination of traits and scales gives 3x3 different questions. • All 9 questions have been presented to a British sample of 485 people • That means that differences in responses can not be due to sampling fluctuation but only question format college titel en nummer

  6. It matters for the distributions college titel en nummer

  7. It matters for the correlations college titel en nummer

  8. Research questions • Using the MTMM design one can see that there are random and systematic errors • We have many examples of the same kind • How large are the different errors ? • Can we derive general rules with respect to the effects of question characteristics ? college titel en nummer

  9. Content of this presentation • 1. Definition of reliability and validity in terms of • random and systematic error • 2. SEM approaches to estimation of reliability and • validity • 3 The effects of survey characteristics on reliability • and validity • 4 Prediction of data quality by SQP • 5 Conclusions college titel en nummer

  10. Definition of reliability and validity Definition of reliability and validity in terms of random and systematic error

  11. The Classical test model • y is the observed variable • e is random measurement error across persons and occasions • t is called the true score: t = y - e • The strength of the relationship between t and y is called the reliability • The reliability = 1 - var(e) college titel en nummer

  12. Difference due to the method used • The strength of the relationship between t and f is called the validity: • The Validity = 1 - var (m) college titel en nummer

  13. Measurement model for two traits , same method college titel en nummer

  14. SEM Approach to estimation of survey errors Different Designs and Models

  15. Different designs to estimate reliability and validity Different designs to estimate the reliability and validity: • test retest design • quasi simplex design • Split ballot design • Evaluation of validity by nomologic network • Classic MTMM design • SB MTMM design • RSB MTMM design college titel en nummer

  16. Test retest design and model • The same observation is done twice with the same method • Assuming that there is no difference between f and t • the relationship between f and y is equal to the square root of the reliability • if ri1 = ri2 then r(yi1yi2) = ri12 • so the reliability = r(yi1yi2) • but college titel en nummer

  17. Criticism on the Test retest model • This model reduces only to the test retest model if • 1.no change in opinion between the first and the second measurement • 2.no memory effects • 3.no method effects • 4.equal reliability for the different measures of the same trait. • How to combine 1 & 2 ? college titel en nummer

  18. The quasi simplex design and model • In this design the same observation is repeated 3 times with the same method • Now the variable on interest can change and the reliability coefficient can still be estimated • but college titel en nummer

  19. Criticism on the Quasi simplex model • It is a lag one model. An effect of f1 on f3 is impossible and will lead to completely different estimates of the reliabilities (Coenders) • All temporary effects, not present on the next moment are included in the measurement error term which will be overestimated (van der Veld) college titel en nummer

  20. Split Ballot Design • In many studies (Schuman and Presser 1981) the sample is randomly split up in two or more groups • Each group is confronted with a different form of the question • If the score on f is known from other sources the error = yi - fi and the estimate of bias = mean(y) - mean(f) • If the score on f is unknown one can obtain an estimate of the relative bias: • rel. bias = mean(ym1) - mean (ym2) college titel en nummer

  21. Criticism on the Split ballot design • If the relative bias is significantly different from zero then we know that at least one of the methods in not valid but we don’t know which one. • In order to see which method is better, the relationships with variables, that should correlate highly with the variable of interest, are observed • The method that generates the highest correlation is the most valid measure • but college titel en nummer

  22. Criticism of the construct validity approach • Construct validity is normally evaluated by estimating the correlation with other correlated variables • ry1j,xi = r1j v1j r • and • ry2j,xi = r2j v2jr • The difference in correlation can come from the reliability or the validity • So v1j and v2j should be compared not the correlations college titel en nummer

  23. Classical MTMM design and model • Andrews (1984) suggested the Classical MTMM model for the design with 3 traits and 3 methods of Campbell and Fiske (1956) • In that case the reliability and validity coefficients for all 9 questions and the correlations between the traits can be estimated using a factor model • Saris and Andrews (1991) suggested the True Score MTMM model college titel en nummer

  24. Factor model representation college titel en nummer

  25. Criticism on the MTMM design • Respondents have to answer three times a question about the same topic • After 20 minutes most people have forgotten what they answered before (Van Meurs and Saris) • But this requires long questionnaires • The response burden is high: it may be that people become less concentrated • It is also possible that they learn how to answer these questions college titel en nummer

  26. The Split Ballot MTMM design • The Split Ballot MTMM design has been developed by Saris, • The estimation and testing are described in Saris, Satorra and Coenders (2004) • The idea is that the sample is split up randomly in two or more groups • Each group gets only 2 forms of the questions • Using different groups all methods will be used at least once college titel en nummer

  27. Covariance Matrix for a 2group design Method 1 Method 2 Method 3 Method 1 Sample 1 Method 2 none Sample 2 Method 3 Sample 1 Sample 2 Samples 1 + 2 Missing data and Identification in the two groups design • Although one block of variables is completely missing by design all the parameters of the model can still be estimated unless: • The correlations between the traits are zero or • The correlations are exactly equal to each other college titel en nummer

  28. Criticism of the SB-MTMM design • Large samples are needed to get estimates with the same precision as the classical MTMM design • A more serious problem of all MTMM designs is that the method and occasion effects are confounded. • We can not separate these effects without an exact repetition of the same method at two occasions. college titel en nummer

  29. The Repeated SB-MTMM design • From this model follows: • (Y1jk, Y2jk) = r1jkv1jk (F1,F2)v2jkr2jk + r1jkm1jkm2jkr2jk + r1jko1jko2jkr2jk • To make a distinction between the effect of the methods and the occasions exact repetition is required college titel en nummer

  30. Results of the MTMM experiment on satisfaction college titel en nummer

  31. The explanation of the different correlations for satisfaction • For method 1(a 4 point scale) this correlation was .481 • For method 2 (an 11 point scale) this correlation was .626 • The correlation between the observed variables can be predicted with the MTMM models as follows: • r(Y1j,Y2j) = r1jv1j r(f1,f2)v2jr2j + r1jm1jm2jr2j college titel en nummer

  32. Alternative models • Alternative models for correlated errors in stead of method effects: • Multiplicative models • Memory effect models • Acquiescence model • Variation in response functions model • Corten et al (2002) and Saris and Albers (2003) have shown that the MTMM model is better than the other models. college titel en nummer

  33. Conclusions • The best design and model to estimate reliability and validity is the MTMM, preferably the RSB MTMM design and model • The results cannot be generalized to other measures • This approach is rather expensive if all researchers should use this model to correct for measurement error • Therefore an alternative has been developed. college titel en nummer

  34. Prediction of Survey quality Meta analysis of MTMM experiments

  35. An alternative: Meta analysis of MTMM experiments • Andrews (1984) performed a meta analysis of MTMM experiments • The MTMM studies provide estimates of the reliability and validity of questions • The questions can be coded on design decisions • Regressing the quality criteria on the design decisions based on a large number of questions provides estimates of the effects of the design decisions on the question quality college titel en nummer

  36. Meta analysis • C represents the score on a quality criterion • Dij represents the dummy variables for the jth nominal variable. • All dummy variables have the value zero unless the specific characteristic applies for a question. • Continuous variables, like Ncat, have not been categorized • The intercept (a) is the reliability or validity of the instruments if all variables have a score of zero. college titel en nummer

  37. The most recent cross national meta analysis • In total, 87 MTMM studies have been used containing 1067 survey items. • They come from : • Andrews (1984) and Rogers, Andrews and Herzog (1989) in the US. • Költringer (1995) in Austria • Scherpenzeel and Saris (1997) in the Netherlands • Billiet and Waege in Belgium college titel en nummer

  38. The most recent cross national meta analysis • The results have been presented in full in Saris, Van der Veld and Gallhofer (2004) • The explained variance for reliability = .47 • The explained variance for validity = .61 • Some results of the analysis are presented college titel en nummer

  39. Meta analysis college titel en nummer

  40. Meta analysis college titel en nummer

  41. Meta analysis college titel en nummer

  42. Meta analysis college titel en nummer

  43. Meta analysis college titel en nummer

  44. Meta analysis college titel en nummer

  45. Meta analysis college titel en nummer

  46. Meta analysis college titel en nummer

  47. Meta analysis college titel en nummer

  48. Meta analysis college titel en nummer

  49. Meta analysis college titel en nummer

  50. Conclusion • The meta analysis of the MTMM experiments gives an explanation of the variation in reliability and validity on the basis of the choices made in designing a survey (question) • This result contains all the present information that is available from MTMM experiments • This is a temporary result because new data are collected. college titel en nummer

More Related