Designing Monte Carlo Simulation Studies

1. Designing Monte Carlo Simulation Studies Xitao Fan, Ph.D. Chair Professor & Dean Faculty of Education University of Macau

2. Getting Involved in Monte Carlo Simulation Fan, X. (2012). Designing simulation studies. In H. Cooper (Ed.), Handbook of Research Methods in Psychology, Vol. 2 (pp. 427-444). Washington, DC: American Psychological Association. Fan, X., Felsovalyi, A., Sivo, S. A., & Keenan, S. (2002) SAS for Monte Carlo studies: A guide for quantitative researchers. Cary, NC: SAS Institute, Inc.

3. Peugh, J., & Fan, X. (In press). Enumeration index performance in generalized growth mixture models: a Monte Carlo test of Muthén’s (2003) hypothesis. Structural Equation Modeling. Peugh, J., & Fan, X. (In press). Modeling unobserved heterogeneity using latent profile analysis: A Monte Carlo simulation. Structural Equation Modeling. Peugh, J., & Fan, X. (2012). How well does growth mixture modeling identify heterogeneous growth trajectories? A simulation study examining GMM’s performance characteristics. Structural Equation Modeling, (19), 204-226. Fan, X., & Sivo, S. A. (2009). Using goodness-of-fit indices in assessing mean structure invariance. Structural Equation Modeling, 16, 1-16. Fan, X. & Sivo, S. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42, 509-529. Sivo, S. A., Fan, X., Witta, E. L., & Willse, J. T. (2006). The search for "optimal" cutoff properties: Fit index criteria in structural equation modeling. Journal of Experimental Education, 74, 267-288. Getting Involved in Monte Carlo Simulation

4. Fan, Xitao, & Fan, Xiaotao. (2005). Power of latent growth modeling for detecting linear growth: Number of measurements and comparison with other analytic approaches. Journal of Experimental Education, 73, 121-139. Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indices to misspecified structural or measurement model components: Rationale of two-index strategy revisited. Structural Equation Modeling, 12, 343-367. Fan, Xitao, & Fan, Xiaotao. (2005). Using SAS for Monte Carlo simulation research in structural equation modeling. Structural Equation Modeling, 12, 299-333. Sivo, S., Fan, X., & Witta, L. (2005). The biasing effects of unmodeled ARMA time series processes on latent growth curve model estimates. Structural Equation Modeling, 12, 215-231. Fan, X. (2003). Two Approaches for Correcting Correlation Attenuation Caused by Measurement Error: Implications for Research Practice. Educational and Psychological Measurement, 63, 6, 915-930. Fan, X. (2003). Power of latent growth modeling for detecting group differences in linear growth trajectory parameters. Structural Equation Modeling, 10, 380-400. Getting Involved in Monte Carlo Simulation

5. Yin, P., & Fan, X. (2001). Estimating R2 shrinkage in multiple regression: A comparison of different analytical methods. Journal of Experimental Education, 69, 203-224. Fan, X., & Wang, L. (1999). Comparing logistic regression with linear discriminant analysis in their classification accuracy. Journal of Experimental Education, 67, 265-286. Fan, X., Thompson, B, & Wang, L. (1999). The effects of sample size, estimation methods, and model specification on SEM fit indices. Structural Equation Modeling: A Multidisciplinary Journal, 6, 56-83. Fan, X., & Wang, L. (1998). Effects of potential confounding factors on fit indices and parameter estimates for true and misspecified SEM models. Educational and Psychological Measurement, 58, 699-733. Fan, X. & Wang, L. (1996). Comparability of jackknife and bootstrap results: An investigation for a case of canonical analysis. Journal of Experimental Education, 64, 173-189. Getting Involved in Monte Carlo Simulation

6. What Is a Monte Carlo Simulation Study?  “the use of random sampling techniques and often the use of computer simulation to obtain approximate solutions to mathematical or physical problems especially in terms of a range of values each of which has a calculated probability of being the solution” (Merriam-Webster On-Line).  An empirical alternative to a theoretical approach (i.e., a solution based on statistical/mathematical theory)  Increasingly possible because of the advances in computing technology

7. Situations Where Simulation Is Useful Consequences of Assumption Violations Statistical Theory: stipulates what the condition should be, but does not say what the reality would be if the conditions were not satisfied in the data Understanding a Sample Statistic That May Not Have Theoretical Distribution • Many Other Situations • Retaining the optimal number of factors in EFA • Evaluating the performance of mixture modeling in identifying the latent groups • Assessing the consequences of failure to model correlated error structure in latent growth modeling

8. Basic Steps in a Simulation Study Asking Questions Suitable for a Simulation Study • Questions for which no (no trustworthy) analytical/theoretical solutions  Simulation Study Design (Example) • Include / manipulate the major factors that potentially affect the outcome  Data Generation • Sample data generation & transformation  Analysis (Model Fitting) for Sample Data  Accumulation and Analysis of the Statistic(s) of Interest  Presentation and Drawing Conclusions • Conclusions limited to the design conditions

9. An Example: Independent t-test (group variance homogeneity)

10. An Example: Independent t-test (group variance homogeneity)

11. Data Generation in a Simulation Study  Common Random Number Generators * binomial, Cauchy, exponential, gamma, Poisson, normal, uniform, etc. * All distributions are based on uniform distribution  Simulating Univariate Sample Data * Normally-Distributed Sample Data (N ~ , 2) * Non-Normal Distribution: Fleishman (1978): a, b, c, d: coefficients needed for transforming the unit normal variate to a non-normal variable with specified degrees of population skewness and kurtosis. Fleishman, A. I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-531.

12. Data Generation in a Simulation Study  Sample Data from a Multivariate Normal Distribution *matrix decomposition procedure (Kaiser & Dickman, 1962): F: k  k matrix containing principal component factor pattern coefficients obtained by applying principal component factorization to the given population inter-correlation matrix R;  Sample Data from a Multivariate Non-Normal Distribution * Interaction between non-normality and inter-variable correlations * Intermediate correlations using Fleishman coefficients (Vale & Maurelli, 1983) * Matrix decomposition procedure applied to intermediate correlation matrix Kaiser, H. F., & Dickman, K. (1962). Sample and population score matrices and sample correlation matrices from an arbitrary population correlation matrix. Psychometrika, 27, 179-182 Vale, C. D., & Maurelli, V. A. (1983). Simulating multivariate nonnormal distributions. Psychometrika, 48, 465-471.

13. Checking the Validity of Data Generation Procedures  Example: Multivariate non-normal sample data (three correlated variables)

14. From Simulation Design to Population Data Parameters  It may take much effort to obtain population parameters – t-test example

15. From Simulation Design to Population Data Parameters  Latent growth model example

16. From Simulation Design to Population Data Parameters  Latent growth model example

17. Accumulation and Analysis of the Statistic(s) of Interest Accumulation: Straightforward or Complicated * Typically, not an automated process * Statistical software used * Analytical techniques involved * Type of statistic(s) of interest, etc. Analysis * Follow-up data analysis may be simple or complicated * Not different from many other data analysis situations

18. Presentation and Drawing Conclusions Presentation * Representativeness & Exceptions * Graphic Presentations * Typical: table after table of results – No one has the time to read the tables!  Drawing Conclusions * Validity & generalizability depend on the adequacy & appropriateness of simulation design * Conclusions must be limited by the design conditions and levels.