1 / 19

Methods Matters

Methods Matters. Grover J. (Russ) Whitehurst Director Institute of Education Sciences United States Department of Education. Methods Matter: High Stakes. Make-or-Break Exams Grow, But Big Study Doubts Value

steinj
Download Presentation

Methods Matters

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Methods Matters Grover J. (Russ) Whitehurst Director Institute of Education Sciences United States Department of Education

  2. Methods Matter: High Stakes Make-or-Break Exams Grow, But Big Study Doubts Value “Rigorous testing that decides whether students graduate, teachers win bonuses and schools are shuttered, an approach already in place in more than half the nation, does little to improve achievement and may actually worsen academic performance and dropout rates, according to the largest study ever on the issue.”

  3. Methods Matter: High Stakes New Ammunition for Backers of Do-or-Die Exams “Two new studies make the case that do-or-die exams -- which decide whether students graduate, teachers are dismissed or schools are shut in more than half the states in the nation -- have brought about at least a modicum of academic progress, especially for minority students who may get scant attention otherwise.”

  4. Methods Matter: High Stakes Study Finds Higher Gains in States With High-Stakes Tests "I've had a lot of people reanalyze our data … and each and every one of them have come up with different results.”

  5. Methods Matter • Methods match questions • Description • Association • What works • Why

  6. What Works Deconstructed

  7. What Works: Causal Validity Meets Standards • Randomized controlled trial with no randomization, attrition, or disruption problems • Regression discontinuity study with no comparability, attrition, or disruption problems

  8. Causal Validity Meets Standards with Reservations • Randomized controlled trial with a randomization, attrition, and/or disruption problem • Regression discontinuity study with a comparability, attrition, or disruption problem • Quasi-experimental design with equivalent groups and no problems with attrition or disruption

  9. Causal Validity Does not meet Standards • Anecdotes • Case studies • Pre-post studies • Correlational studies • Value-added studies • Conceptual models • Flawed comparison group studies • Narrative summaries • Meta-analyses of flawed studies

  10. Intervention Fidelity • The intervention contains most of the key characteristics that commonly define it • The author provides evidence of good implementation • The intervention is documented well enough for others to replicate it

  11. Outcome Measures • Face Validity • Reliability • Not over-aligned

  12. People, Settings, and Timing • Samples are from the people (units of interest) and settings that are the target of the intervention • Outcomes are measured at an appropriate time

  13. Statistical Analysis • The analysis is conducted at the same level (for example, students, classes, schools) as the unit of assignment • Accounts for clustering effects (for example, when the intervention is delivered to all students in a classroom)

  14. Statistical Reporting • Findings are reported for most outcome measures • Effect sizes can be calculated

  15. Rigorous Evaluations at IES • Early Reading First National Evaluation • Reading First Impact Study • After-School Programs • Remedial Reading Programs • Teacher Preparation Models • Professional Development Models • Educational Technology Interventions • Interventions for English Language Learners

  16. Rigorous Evaluations at IES • Charter School Strategies • D.C. Choice Program • Even Start • Teacher Induction Programs • Preschool Curriculum • Character and Socialization Interventions

  17. Rigorous Evaluations Planned • Safe and drug-free school programs • Interventions for Adult ESL Students • Many individual grants, including IERI

  18. The Institute of Education Sciences The home of evidence-based education

More Related