1 / 16

Implementation of Randomized Trials in Educational Research

Learn about the design and implementation of randomized trials for conducting effectiveness studies in education, focusing on preventing distortion of programs and populations served. Explore illustrative studies and practical challenges involved in random assignment.

nelly
Download Presentation

Implementation of Randomized Trials in Educational Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Implementation of Randomized Trials David Myers American Institutes for Research Washington, DC Prepared for IES/NCER Summer Research Training Institute, 2008

  2. Context • Conducting effectiveness trials and not efficacy trials • Design and implementation of random assignment should not distort the program/intervention or the population served • Programs should run at capacity

  3. Illustrative Studies: Personal Involvement • Upward Bound • Nationally representative study • 67 sites and more than 3,000 students • Student RA within site • http://www.mathematica-mpr.com/publications/PDFs/upboundimpact.pdf • Workplace Literacy • 3 sites and about 500 adult learners • Learner RA within site • "Addressing Literacy Needs at Work: Implementation and Impact of Workplace Literacy Programs. Final Report." Washington, DC: Mathematica Policy Research, Inc., October 1998

  4. Illustrative Studies: Personal Involvement • 21st Century Community Learning Centers • 12 grantees (elementary school study) • Student RA within site • http://www.mathematica-mpr.com/publications/PDFs/elementaryschools.pdf • NYC Voucher Experiment • 2000 students • Family RA • http://www.mathematica-mpr.com/publications/PDFs/nycfull.pdf

  5. Illustrative Studies: Personal Involvement • Reading Comprehension • 10 districts and 40 schools • School RA • http://www.mathematica-mpr.com/publications/PDFs/readcomp.pdf • Closing the Reading Gap • 32 schools (4 interventions) • Student RA to reading groups within schools • http://ies.ed.gov/ncee/pdf/20084013.pdf

  6. Illustrative Studies: Personal Involvement • Quantum Opportunities • 7 sites and about 1000 students • Student RA within site • http://www.mathematica-mpr.com/publications/PDFs/QOPfinalimpacts.pdf

  7. Practical Problems in Implementation • Technical • Political and Ethical Challenges • Recruitment (not independent of P and E Challenges)

  8. Further Exploration: Technical • Cross-overs • Members of C get into T or something like T (use UB example and James Comer example) • Into T -- straight forward adjustment (AIR, 1996) • Something like T -- ?; affects interpretation • Attrition • Post baseline -- “straight forward” adjustment (e.g., MI) • Before baseline -- problematic, no information • How to fix?????

  9. Further Exploration: Technical • Unbalanced designs • Don’t demand additional recruitment • Large variance in selection probabilities (weights) • “over subscription” didn’t meet expectations (UB example) • Dishonest assignment -- post randomization • Sites don’t tell all students they have been selected for the program (UB example) • Sites ignore RA and move controls into T

  10. Further Exploration: Ethical and Political Challenges • Random assignment isn’t fair • Programs afraid of denying services to students • Is it fair to never give a student a chance? (UB example) • Random assignment will force a program/teacher to serve a different population • Role of stratification to serve the desired mix • Some students (units) must be served • Role of the “wild card” before randomization • Wild cards excluded from analysis

  11. Further Exploration: Ethical and Political Challenges • All “seats” must be filled • Role of the waiting list • Random selection within strata, if needed and desired by program operators • T and C groups analyzed as implemented at initial randomization • When programs believe recruitment will distort the population • Identify “most likely” and “least likely” to serve • Prior to randomization and stratify in analysis

  12. Further Exploration: Ethical and Political Challenges • New treatments/interventions are hard to sell • Core programs vs. supplemental programs • Does the core program align with other curriculum and state assessments? • Concerns about making AYP • Will a supplemental program reduce hours of instruction in a core area such as ELA?

  13. Strategies for Recruitment: Schools • Start with assistant superintendent for instruction (reading, math, science) or someone of similar stature • Quickly develop a relationship with an office and not just an individual -- staff come and go • Determine who needs to approve and to buy into participation • Superintendent -- they may need to go to the school board • Principals • Teachers • Parents and community groups

  14. Strategies for Recruitment: Schools • Be prepared to meet with them • Have a recruitment team • Technical expertise in the design • Be prepared to tell audience why RA is valuable and not in a technical sense! • Deep knowledge about the intervention -- most important • Example of a meeting (next slide)

  15. Irrefutable Evidence

  16. Strategies for Recruitment: Schools • Establish expectations for the researchers and the schools • Minimize legal talk • Will bring in their general counsel and … . • Responsibilities and timelines • Obtaining consent • Data collection • Administering instruments and tests • Answering questions from parents and others

More Related