1 / 37

Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement

Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement. Lisa D. Cota, MS Statistical Analyst Office of University Assessment & Testing Oklahoma State University. Why not NHST?.

michel
Download Presentation

Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement Lisa D. Cota, MS Statistical Analyst Office of University Assessment & Testing Oklahoma State University

  2. Why not NHST? • Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003. Retrieved from http://psycnet.apa.org/journals/amp/49/12/997/ • Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1–20. Retrieved from http://www2.uni-jena.de/svw/metheval/lehre/0405-ws/evaluationuebung/haller.pdf • Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review. doi:10.3758/s13423-013-0572-3 • Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31(1), 65–71. doi:10.1016/j.newideapsych.2011.02.009 • Lykken, D. What’s wrong with psychology anyway? (1991). In Thinking Clearly About Psychology. University of Minnesota Press. • Toomela, A. (2010). Quantitative methods in psychology: Inevitable and useless. Frontiers in Psychology, 1(July), 29. doi:10.3389/fpsyg.2010.00029

  3. Why not NHST? • Assessment should be understandable • NHST: Not easily understood…in fact easily misunderstood • Oakes, 1986; Gigerenzer, 2004 • Hoekstra, Morey, Rouder, & Wagonmaker, 2014 • One problem (of many) with NHST: • Variance Explained ≠ Accuracy

  4. y = 0.5x + 3, R2 = 0.663

  5. y = 0.5x + 3, R2= 0.663

  6. y = 0.5x + 3, R2= 0.663

  7. y = 0.5x + 3, R2= 0.663

  8. Assumptions • Assumptions for independent samples t-test under NHST: • Null hypothesis is true • Observations are independent between and within groups • Population variances are exactly equal • Bivariate IV • Random sampling / random assignment (???) * Continuous DV * DV has a normal population distribution

  9. Observation Oriented Modeling • No meaningless aggregates • Person-centered • No assumption-laden p-values

  10. Observation Oriented Modeling: Ordinal Pattern Analysis

  11. Results

  12. Randomization Test

  13. Results, with chance-value

  14. VALUE Rubric Data

  15. Results

  16. Significant ≠ Meaningful

  17. Significant ≠ Meaningful

  18. Significant ≠ Meaningful

  19. Transparency, Honesty, and Progress

  20. Raw Numbers

  21. Proportions

  22. Compare w. SPSS….

  23. SPSS Legacy Dialog, k Independent Samples

  24. Proportions

  25. Advantages of OOM • Assumption-free • Transparent • Honest

  26. OOM website • “We’ve never actually looked at the data!” • www.idiogrid.com/OOM

More Related