Effect size measures for single-case designs: between-case standardized mean differences

1. Single-Case Intervention Research Training Institute Madison, WI - June, 2018 James E. Pustejovsky pusto@austin.utexas.edu Effect size measures for single-case designs:between-case standardized mean differences

2. Parametric between-case effect sizes • Within-case standardized mean difference is not on the same scale as SMD from between-groups designs (e.g., between-subjects randomized trial). • Shadish, Rindskopf, & Hedges (2008) asked: Can we estimate a SMD based on the data from a single-case design that IS in the same metric a SMD from a between-groups design? • Why do this? (Shadish, Hedges, Horner, & Odom, 2015) • Translation of single-case research for researchers who work primarily with between-groups designs. • Comparison of results from single-case studies and between-groups studies, for purposes of understanding the utility and limitations of each type of design. • Synthesis involving both single-case and between-groups designs.

3. Between-case SMD • What is the SMD from a between-groups experiment? • These quantities can be estimated from single-case data using a hierarchical model that describes variation within and between participants. • But we’ll need to have a sample of multiple participants (bare minimum of 3, more for more complex models).

4. Estimating between-case SMDs: The broad strategy: • Develop a hierarchical model that describes a) the functional relationship for each case and b) how the outcome and functional relationship vary across cases. • Use the hierarchical model to imagine a hypothetical between-subjects experiment with the same population of participants, same treatment, same outcomes. • Calculate the between-case SMD for the hypothetical experiment.

5. Estimating BC-SMDs: Basic model Hedges, Pustejovsky, and Shadish proposed BC-SMD estimators for a basic hierarchical linear model • HPS (2012): Treatment reversal (ABAB) designs • HPS (2013): Multiple baseline/multiple probe designs Assumptions: • Baseline is stable (no baseline trend). • Intervention effect is immediate (no intervention-phase trend). • The outcome is normally distributed around mean level for each case, with variance σ2. • Within-case errors follow a first-order auto-regressive process (serial dependence). • The baseline level for each case is normally distributed, with variance τ2. • The treatment effect is constant across cases.

6. Estimating BC-SMDs: Basic model • Moment estimation • Web app: https://jepusto.shinyapps.io/scdhlm/ • R package: http://jepusto.github.io/getting-started-with-scdhlm • SPSS macro: http://faculty.ucmerced.edu/wshadish/software/software-meta-analysis-single-case-design/dhps-version-march-7-2015 • Shadish, Hedges, and Pustejovsky (2014) includes worked examples • Restricted maximum likelihood estimation (recommended) • Web app: https://jepusto.shinyapps.io/scdhlm/ • R package: http://jepusto.github.io/getting-started-with-scdhlm • Both methods produce estimates of BC-SMD (corrected for small-sample bias) and accompanying standard error.

7. Rodriguez & Anderson (2014) example (using REML estimation)

8. More flexible models for BC-SMDs • Pustejovsky, Hedges, and Shadish (2014) extend the basic model to allow for less restrictive assumptions: • Variability of individual treatment effects • Baseline time trends (constant or varying across participants) • Time trends in treatment phase (constant or varying across participants) • More flexible models require more cases • For models with time trends, also need to specify a focal follow-up time

9. Barton-Arwood, Wehby, & Falk (2005). Reading instruction for elementary-age students with emotional and behavioral disorders: Academic and behavioral outcomes

10. Barton-Arwood, Wehby, & Falk (2005)Effect size calculations • Allow for baseline and intervention time trends, both varying across cases. • Focal follow-up time after 13 sessions. • BC-SMD estimate of 0.91 (SE = 0.93). • Substantial heterogeneity in intervention time trends.

11. Limitations of between-case SMD • Describes an average effect across a set of cases • Conceals potential individual heterogeneity • Inherent consequence of comparability with between-groups effect sizes. • Technical limitations • Only available for treatment reversal (ABAB) and multiple baseline/multiple probe across participant designs. • Requires at least 3 participants, preferably more. • Assumes normally distributed, interval-scale outcomes. • More work needed on evaluating model selection, model fit • Use between-case effect sizes as a complement to (not a replacement for) within-case effect size measures

12. References Beretvas, S. N., & Chung, H. (2008). An evaluation of modified R2-change effect size indices for single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 120–128. doi:10.1080/17489530802446328 Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Research Design and Analysis: New Directions for Psychology and Education (pp. 187–212). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: a quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24(2), 120–138. doi:10.1016/S0891-4222(03)00014-3 Campbell, J. M., & Herzinger, C. V. (2010). Statistics and single subject research methodology. In D. L. Gast (Ed.), Single Subject Research Methodology in Behavioral Sciences (pp. 417–450). New York, NY: Routledge. Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied Behavioral Science, 20(1), 71–79. doi:10.1177/002188638402000113 Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224–239. doi:10.1002/jrsm.1052 Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2013). A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods, 4(4), 324–341. doi:10.1002/jrsm.1086 Kahng, S., Iwata, B. a, & Lewin, A. B. (2002). Behavioral treatment of self-injury, 1964 to 2000. American Journal of Mental Retardation : AJMR, 107(3), 212–221. doi:10.1352/0895-8017(2002)107<0212:BTOSIT>2.0.CO;2 Maggin, D. M., Swaminathan, H., Rogers, H. J., O’Keeffe, B. V, Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49(3), 301–321. doi:10.1016/j.jsp.2011.03.004 Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P., Thompson, M., Behrens, G. A., … Doolabh, A. (2000). A meta-analysis of positive behavior support. In R. Gersten, E. P. Schiller, & S. Vaughan (Eds.), Contemporary Special Education Research: Syntheses of the Knowledge Base on Critical Instructional Issues (pp. 137–178). Mahwah, NJ: Lawrence Erlbaum Associates. Pustejovsky, J. E., Hedges, L. V, & Shadish, W. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39(5), 368–393. doi:10.3102/1076998614547577 Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20(3), 342–359. doi:10.1037/met0000019 Pustejovsky, J. E., & Swan, D. M. (2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50(3), 365–380. doi:10.1080/00273171.2015.1014879 Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 188–196. doi:10.1080/17489530802581603 Shadish, W. R., Hedges, L. V, & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. doi:10.1016/j.jsp.2013.11.005 Shadish, W. R., Hedges, L. V, Horner, R. H., & Odom, S. L. (2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research. Washington, DC. Retrieved from http://ies.ed.gov/ncser/pubs/2015002/ Rodriguez, B. J., & Anderson, C. M. (2014). Integrating a social behavior intervention during small group academic instruction using a total group criterion intervention. Journal of Positive Behavior Interventions, 16(4), 234–245. doi:10.1177/1098300713492858