where are we going l.
Skip this Video
Loading SlideShow in 5 Seconds..
Where are we going? PowerPoint Presentation
Download Presentation
Where are we going?

Loading in 2 Seconds...

play fullscreen
1 / 60

Where are we going? - PowerPoint PPT Presentation

  • Uploaded on

Where are we going?. What to do if no RCT, RD, ITS or sophisticated matching is possible? We describe and analyze principle of pattern matching to improve the basic workhorse design by a feature other than high quality matching

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Where are we going?' - anika

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
where are we going
Where are we going?
  • What to do if no RCT, RD, ITS or sophisticated matching is possible?
  • We describe and analyze principle of pattern matching to improve the basic workhorse design by a feature other than high quality matching
  • Illustrate what you might do if there is no pretest at all and so not even a work-horse design is possible
  • Before that, a bit of a summary
yesterday 1
Yesterday 1
  • Do not match from extremes unless forced to
  • Then match using a reliable set of measures, moving to propensity score framework, latent variables, or statistical reliability-adjustments to handle unreliability
  • But you will still have problem of possible specification error!
yesterday 2
Yesterday 2
  • In selecting non-equivalent control group:
  • Use local focal matching to reduce the degree of initial non-comparability though you cannot expect total non-comparability
  • Sometimes the control group so formed will not differ from what would have been achieved with random assignment, at least on observables
  • At other times, the initial group non-equivalence will be reduced for when you come to do statistical analysis to “control” initial differences
yesterday 3
Yesterday 3
  • When there is initial non-equivalence on observables:
  • Theory and empiricism about the constructs (covariates) in the “true”selection model helps, as does careful measurement thereof
  • Measures in multiple other domains helps too,
  • Unclear whether ANCOVA or propensity scores do better--Shadish et al, Glazerman et al.--though demographic variables alone and Heckman IV models do not fare well. Propensity scores preferred on theory alone.
  • You can never be sure of the final causal conclusion, though
  • In designing research you do well to avoid the workhorse if you can, though it is modal in current educational practice
  • Can you add prior pretest waves?
  • Can you add any of the other design elements, some mentioned in our ITS discussion and others we discuss today
  • How can you design your way out of reliance on a simple design with non-equivalent groups and pretest and posttest?
when there is no pretest on the same scale as outcome
When there is no Pretest on the same scale as Outcome
  • Do randomized experiment--Abcedarian, Perry-Preschool, Head Start, Early Head Start, CLIO, Even Start and Sesame Street
  • If not possible, do everything possible to make control group focally local
  • Add design elements to rule out alternative explanations of a possible causal relationship
  • Here’s an example
minton s dissertation
Minton’s Dissertation
  • Object: Evaluate Sesame St in 1st year
  • Problem 1: Program already launched
  • Problem 2: No pretest possible
  • Problem 3: No money for original data collection
  • Setting: One kindergarten in NJ that built SS into its day and that has records on children and their families plus annual PPV assessment
question 1 what control group is possible
Question 1: What control group is possible?
  • What control group to find, given program was very popular in its first year.
  • Why is popularity a problem?
  • Neighborhood kids who did not view
  • Next-door kids of same age who not view?
  • Older siblings in general
  • Older sibs attending same kindergarten within last N years
  • Older sibs attending same kindergarten last 2 years

Minton (1975) Sesame Street Study - 1

Achievement scores

Older control siblings

Younger Sesame Street siblings


Minton (1975) Sesame Street Study - 2

Letter skills

Achievement scores

Non-letter PPVT skills

Older control siblings

Younger Sesame Street siblings


Minton (1975) Sesame Street Study - 3

Letter skills – high viewers

Letter skills – low viewers

Achievement scores

Non-letter skills–low viewers

Non-letter skills–high viewers

Older control siblings

Younger Sesame Street siblings

what has happened here
What has happened here?
  • Single causal hypothesis of SS effective made to have multiple data implications
  • These are meant to rule out alternative hypotheses and not to recreate same bias
  • These implications here in the form of a difference in difference in differences
  • Collect data and test hypothesis
another example
Another Example
  • How the Introduction of TV affected Library Circulation

Parker et al. (1966) Effects of TV - 1

Short interrupted time series

Library circulation per capita

Fiction book circulation






Parker et al. (1966) Effects of TV - 2

Short interrupted time series with control

Library circulation per capita

Early TV communities

Late TV communities






Parker et al. (1966) Effects of TV - 3

ITS with switching replication

1949 interruption

Library circulation per capita

1953 interruption






Parker et al. (1966) Effects of TV - 4

ITS with switching replication and control



Library circulation per capita







what has happened here18
What has happened here?

Combine an ITS with non-equivalent DVs and switching replications

What alternative interpretations can you come up with?

How plausible are these?

  • Have we seen this before with RD and ITS?
  • One general causal hypothesis has multiple implications in the data
  • Predicted hypothesis as multiple differences of differences; as higher order interactions
reynolds and west s 1987 ask for the sale experiment
Reynolds and West’s (1987) “Ask for the Sale” Experiment

From all stores selling lottery tickets, some stores volunteered (or not) to post a sign reading “Did we ask you if you want a Lottery ticket? If not, you get one free”. So this is a basic nonequivalent control group design, with the control matched on zip code, store chain, and pretest ticket sales.

NR O1 X O2


NR O1 O2

the outcome of the basic design
The Outcome of the Basic Design

But there might be many reasons besides treatment that caused treatment group sales to rise.

adding a nonequivalent dv
Adding a Nonequivalent DV
  • They added three nonequivalent dependent variables, showing that the intervention increased ticket sales but not sales of gas, cigarettes, or grocery items.
adding multiple baselines recasting as its design
Adding Multiple Baselines - recasting as ITS Design
  • They located some stores in which the treatment was initiated later than in other stores, or initiated and then removed, and found that the outcome tracked the introduction of treatment over time while sales in the matched controls remained unchanged
adding multiple pretests and posttests
Adding Multiple Pretests and Posttests
  • They added multiple pretests and posttests by examining mean weekly ticket sales for four weeks before and four weeks after the treatment started.
the point is
The Point is:
  • To use the choice of additional design elements to rule out more alternative interpretations, hopefully all that can be currently identified
  • The goal is ruling out plausible alternative interpretations, and it can also be reached via keeping the pattern of results constant but varying the number and type of comparisons involved

Main NAEP 4th grade math scores

by year and proficiency standards


Main NAEP 4th grade math scores by year:

Public and Other Private schools

  • This pattern matching strategy requires:
  • Clear causal hypothesis - relevance to discontinuity
  • Careful measurement - reliability and ceiling or floor effects
  • Large samples (or large effects) because hypothesis is of a complex statistical interaction
  • How lucky Minton was!
examples from you of the basic work horse design
Examples from you of the Basic Work-Horse Design
  • Let us take some from you and see if they can be improved by adding design elements.
design elements to be combined assignment
Design Elements to be combined: Assignment
  • Random Assignment
  • Cutoff-Based Assignment
  • Researcher-controlled Matching -- of many kinds in econometric literature
design elements to be combined treatments
Design Elements to be combined:Treatments
  • Switching Replications
  • Reversed Treatments
  • Removed Treatments
  • Repeated Treatments
design elements to be combined measurement
Design Elements to be combined: Measurement
  • Single Pretest
  • Pretest Time Series
  • Proxy Pretests
  • Retrospective Pretests
  • Moderators with predicted Interactions
  • Measuring Threats to Validity
design elements to be combined comparison groups
Design Elements to be combined: Comparison Groups
  • Single Non-Equivalent Groups
  • Multiple Non-Equivalent Groups
  • Twins/Siblings
  • Cohorts
  • Other Focal, Local Comparison Groups
golden rules 1
Golden Rules (1)
  • You can’t put right through statistics what you have done wrong by design
  • Statistical adjustments work better the less non-equivalence there is to adjust away in the first place
  • Since the work horse is so prevalent but so problematic, how can we complexify the design through adding design elements
golden rules 2
Golden Rules (2)
  • First, Do an experiment; if not
  • Do Regression-discontinuity study. If not,
  • Do ITS with some sort of a comparison series. If not
  • Do study combining multiple design element, preferably with focal local intact controls, case matching on many covariates, reintroduction of treatment at new time, non-equivalent DVs, etc.
golden rules 3
Golden Rules (3)

Don’t be bamboozled by fancy models in Greek clothing. Always translate them into structural design elements before evaluating their likely validity. That will reveal what you have got

  • Remember you only control for the reliably measured part of any construct, not the construct itself.
evaluation formative
Evaluation, formative

On a scale from 1 to 6, with 6 being high, please rate the following and then indicate how you would improve what we did.

Contact with Valerie about the workshop



Curriculum Content

Curriculum Relevance to your current or anticipated work

Quality of Instruction

Any other Suggestions for Improvements?

grant opportunities at the institute of education sciences

Grant Opportunities at the Institute of Education Sciences

Allen Ruby, Ph.D.

Associate Commissioner

Policy & Systems Division

National Center for Education Research

take away
Take Away
  • IES has a small number of education grant programs
    • Matrix of topics/goals within grant programs
    • Range of work: exploration, development, evaluation
    • Focus is on improving student outcomes
    • Competitions are held twice a year (6/24/10 and 9/16/10)
    • Competitive peer review using an absolute standard
  • Take advantage of available information
    • http://ies.ed.gov (Request For Applications, abstracts of projects, webinars, resources for researchers webpage)
    • Program officer (listed in RFA)

IES Structure

National Board for Education Sciences

Office of the Director

Standards & Review

National Center for Education Statistics

National Center for Education Evaluation

National Center for Education Research

National Center for Special Ed Research

overall research objectives
Overall Research Objectives
  • Develop or identify education interventions (practices, programs, policies and approaches) that enhance academic achievement and that can be widely deployed
  • Identify what does not work and thereby encourage innovation and further research
  • Understand the processes that underlie variations in the effectiveness of education interventions
final outcomes of interest are for students
Final Outcomes of Interest are for Students


  • School readiness
  • Developmental outcomes for infants and toddlers with disabilities

Kindergarten through Grade 12

  • Academic outcomes in reading, writing, math and science
  • Behaviors, interactions, and social skills that support learning in school and successful transitions to post-school opportunities
  • High school graduation
  • Functional skills for independent living of students with disabilities

Postsecondary: enrollment, persistence, and completion

Adult Education: reading, writing, and math (basic, secondary, and EL)

research and research training grant programs
Research and Research Training Grant Programs
  • Education Research Grant Programs
  • Special Education Research Grant Programs
  • Statistical and Research Methodology in Education
  • Evaluation of State and Local Education Programs and Policies
  • Postdoctoral Research Training Grant Programs
  • National Research and Development Centers
education research grants programs 84 305a
Education Research Grants Programs (84.305A)
  • Reading and Writing
  • Mathematics and Science Education
  • Cognition and Student Learning
  • Teacher Quality (Reading & Writing; Math & Science)
  • Social and Behavioral Context for Academic Learning
  • Education Leadership
  • Education Policy, Finance, and Systems
  • Postsecondary Education
  • English Learners
  • Early Learning Programs and Policies
  • Education Technology
  • Adult Education
  • Organization and Management of Schools and Districts
  • Analysis of Longitudinal Data to Support State & Local

Education Reform

special education research programs 84 324a
Special Education Research Programs (84.324A)
  • Early Intervention and Early Childhood Special Education
  • Reading, Writing, and Language Development
  • Mathematics and Science Education
  • Social and Behavioral Outcomes to Support Learning
  • Cognition and Student Learning in Special Education
  • Professional Development for Teachers and Related Service Providers
  • Special Education Policy, Finance, and Systems
  • Transition Outcomes for Special Education Secondary Students
  • Autism Spectrum Disorders
5 research goals
5 Research Goals
  • Exploration: examine relationship between malleable factors and education outcomes
  • Development and Innovation: iteratively develop new or improved education interventions
  • Efficacy & Replication: evaluate the efficacy and feasibility of an intervention
    • Efficacy Follow-up Studies
  • Scale-up Evaluation: evaluate the impact & feasibility of interventions at scale
    • Scale-up Follow-up Studies
  • Measurement: develop and validate assessments
  • Explore malleable factors that are associated with better student learning and education outcomes
    • Malleable factor: something that can changed by the education system - student, teacher, or school characteristics, or an education program or policy
  • Explore factors that mediate or moderate the relationship between malleable factors and student outcomes
  • Small primary data studies, secondary analyses, and meta-analyses
development and innovation
Development and Innovation
  • Develop new interventions (e.g., instructional practices, curricula, teacher professional development, policies)
  • Demonstrate the feasibility of the intervention for implementation in an authentic education delivery setting
  • Collect pilot data on promise of intervention to achieve intended outcomes
efficacy and replication
Efficacy and Replication
  • Test whether or not fully developed interventions are effective under specified conditions and with specific types of students
  • Take place under supportive conditions , e.g. homogenous sample, high assistance
  • Studies using random assignment to intervention and comparison conditions are preferred where feasible
  • New this year: Efficacy follow-up studies
scale up evaluation
Scale-up Evaluation
  • Test whether interventions are effective when implemented under typical conditions.
  • As implemented by practitioners and with sufficiently diverse samples to support generalizability.
  • Studies using randomized assignment to treatment and comparison conditions are preferred whenever they are feasible.
  • Added this year: Scale-up follow-up studies
  • Develop and validate assessments or other measurement tools
    • Typically to be used by practitioners (e.g., screening, progress monitoring, and outcome assessment) but can also be for researcher use
    • Validation of non-student measures must involve student outcomes (e.g. Teacher Quality)
    • Program specific, e.g., cost-accounting under Education Policy, Finance, and Systems
  • Not for evaluating an assessment used as an intervention
  • The measure is the primary product
maximum grant duration and typical size
Maximum Grant Duration and Typical Size

Grants include both direct and indirect costs

  • Exploration: 2 years, $100,000 to $350,000 per yr 10%
    • Up to 4 years & $400,000 for primary data collection
  • Development: 3 years, $150,000 to $500,000 per yr 50%
  • Efficacy: 4 years, $250,000 to $750,000 per yr 26%
    • Follow-up: 3 years, $150,000 to $400,000 per yr
  • Scale-up: 5 years, $500,000 to 1.2 million per yr 2%
    • Follow-up: 3 years, $250,000 to $600,000 per yr
  • Measurement: 4 years, $150,000 to $400,000 per yr 12%
information sources
Information Sources
  • IES website: http://ies.ed.gov
  • Obtain IES Request for Applications
    • Describes type of research that can be done and substance of what to include in application
    • http://ies.ed.gov/funding
  • IES Grants.Gov Application Submission Guide
    • Step by step instructions on how to apply
    • http://ies.ed.gov/funding
  • Obtain Application from and Submit Application to
    • www.grants.gov
    • 84.305A (regular education) & 84.324A (special education)
  • Initial IES Contact (feel free to contact)
    • Allen.Ruby@ed.gov
  • Handout includes information covered in this presentation