Investing in innovation i3 fund evidence evaluation webinar
Download
1 / 42

- PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

Investing in Innovation (i3) Fund Evidence & Evaluation Webinar. June 30, 2011. Note: These slides are intended as guidance only. Please refer to the official documents published in the Federal Register . Agenda. Evidence Standards—eligibility requirement Requirements Review process

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - phineas


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Investing in innovation i3 fund evidence evaluation webinar l.jpg

Investing in Innovation (i3) Fund Evidence & Evaluation Webinar

June 30, 2011

Note: These slides are intended as guidance only. Pleaserefer to the official documents published in the Federal Register.


Agenda l.jpg
Agenda Webinar

  • Evidence Standards—eligibility requirement

    • Requirements

    • Review process

  • Independent Evaluation—program requirement

    • Guidance on qualities of high-quality evaluation plans

  • Questions & answers

    • Live—submission via the Webinar chat function

    • Post-webinar: E-mail to [email protected]


Evidence standards eligibility requirement l.jpg
Evidence Standards— WebinarEligibility Requirement


Evidence eligibility requirements are specific to the type of grant l.jpg
Evidence Eligibility Requirements Are Specific to the Type of Grant

  • An application must meetthe applicable evidence requirements before an award is made

  • Any application failing to meet the applicable evidence requirements is noteligible for a grant award, regardless of its score on the Selection Criteria

  • Any application failing to meet the applicable evidence requirements will notbe considered for a different type of i3 grant award


Scale up grants require strong evidence of effectiveness l.jpg
Scale-up Grants: Require “Strong Evidence” of Effectiveness

  • High internal validity of the evidence

    • Studies designed and implemented in ways that support causal inference

  • High external validity of the evidence

    • Studies based on a sufficient representation of participants and settings that the findings support scaling

  • Minimum size of evidence base

    • More than one well-designed and well-implemented experimental or quasi-experimental study

      OR

    • One large, well-designed and well-implemented multi-site randomized controlled trial


Caution not all associations support causal inferences l.jpg

(Mis-) EffectivenessInterpretive Statement: Based on the evidence, early care & education programs should reduce pupil teacher ratios in order to lower the expulsion rate.

Problem: There are competing explanations for why expulsions increase with increases in pupil-teacher ratios

Caution: Not All Associations Support Causal Inferences


Validation grants require moderate evidence of effectiveness l.jpg
Validation Grants: Require “Moderate Evidence” of Effectiveness

  • Validity of the evidence

    • High internal validity and moderate external validity or

    • High external validity and moderate internal validity

  • Minimum size of evidence base

    • At least one well-designed experimental or quasi-experimental study

      May have small sample sizes OR may fail to demonstrate equivalence between the intervention & comparison groups, but has no other major flaws

      or

    • A correlational study with strong statistical controls for selection bias and for discerning the influence of other potential confounds


Development grants require evidence to support the proposed intervention l.jpg
Development Grants: Require Evidence to Support the Proposed Intervention

  • Theoretical support for the proposed practice, strategy, or program that is based on research findings or reasonable hypotheses, including related research or theories in education and other sectors and

  • Some empirical evidence of the promise of the proposed practice, strategy or program based on prior implementation and evaluation of something similar, albeit potentially on a limited scale or in a limited setting


Evidence standards eligibility review process l.jpg
Evidence Standards— InterventionEligibility Review Process


Responsibility for the evidence eligibility reviews l.jpg
Responsibility for the Evidence Eligibility Reviews Intervention

  • The Institute of Education Sciences (IES) conducts the reviews and reports their findings to OII

  • IES uses What Works Clearinghouse (WWC) evidence standards to judge the causal validity of the evidence

    • IES uses experienced WWC consultants

  • Reviews are limited to evidence identified in the application as that supporting the eligibility requirement


Evidence reviews scale up applications l.jpg
Evidence Reviews: InterventionScale-up Applications

Two focal questions:

1. Does the evidence include a sufficient number and quality of studies?

(1) More than one well-designed and well-implemented experimental study or well-designed and well-implemented quasi-experimental study?

or

(2) One large, well-designed and well-implemented multi-site randomized controlled trial?

2. Does evidence include a reasonable representation of the kinds of participants and settings proposed to receive the practice, strategy, or program under the Scale-up grant to expect the results would generalize?


Evidence reviews validation applications l.jpg
Evidence Reviews: InterventionValidation Applications

Question 1 of 3:

What is the internal validity of the evidence of effectiveness?

High = At least one well-designed and well-implemented experimental or quasi-experimental study supporting the effectiveness of the practice, strategy, or program

Moderate = (a) At least one well-designed experimental or quasi-experimental study, with no major flaws except possibly failure to demonstrate equivalence between the intervention & comparison groups or

(b) At least one correlational study with strong statistical controls for possible selection bias

Low = No study with high or moderate internal validity


Evidence reviews validation cont d l.jpg
Evidence Reviews: InterventionValidation (Cont’d)

Question 2 of 3:

What is the external validity of the evidence?

High = Evidence pertains to the kinds of participants and settings that are the focus of the Validation Grant application

Moderate = Evidence pertains to participants and settings that overlap with but may be more limited than those that are the focus of the application

Low = No study with high or moderate external validity


Evidence reviews validation cont d14 l.jpg
Evidence Reviews: InterventionValidation (Cont’d)

Question 3 of 3:

Does the evidence of effectiveness provided meet one of the following conditions:

1. High internal validity and at least moderate external validity (as defined above)?

or

2. High external validity and at least moderate internal validity (as defined above?


Evidence review development applications l.jpg
Evidence Review: InterventionDevelopment Applications

Two focal questions:

1. Is there a well-reasoned, theoretically supported rationale for the proposed practice, strategy, or program based on research findings or reasonable hypotheses?

2. Is there empirical evidence of the promise that the proposed practice, strategy, or program (or one similar to it) will improve student outcomes (albeit, possibly, based on implementation on a limited scale or in a limited setting compared with the participants and settings proposed to receive the treatment under the Development grant)?


Slide16 l.jpg

What Works Clearinghouse Evidence Standards Interventionhttp://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=19&tocid=1Used to Judge Internal Validity of Evidence


How does the wwc assess research evidence l.jpg
How Does the WWC Assess Research Evidence? Intervention

  • Type of design: does the study design allow us to draw causal conclusions?

  • Strength of data: does the study focus on relevant outcomes and measure them appropriately?

  • Adequacy of statistical procedures: are the data analyzed properly?


Wwc standards apply to causal designs l.jpg

Ineligible Designs Intervention

Anecdotes and testimonials

Case studies

Descriptive

Correlational

WWC Standards Apply to Causal Designs

Eligible Designs

  • Randomized controlled trials (RCTs)

  • Quasi-experimental designs (QEDs)

    Potentially Eligible Designs

  • Regression discontinuity (RDD)

  • Single case (SCD)


Key elements of the wwc rct qed standards l.jpg
Key Elements of the WWC RCT/QED Standards Intervention

Randomization

Yes

No

Attrition

Equivalence

Low

High

Meets Evidence Standards with Reservations

Does Not Meet Evidence Standards

Meets Evidence Standards

Yes

No


Caution 1 rcts with high sample attrition must demonstrate baseline equivalence l.jpg
Caution 1: RCTs with high sample attrition must demonstrate baseline equivalence

Randomization?

Yes

No

Attrition?

High

Low

Baseline Equivalence?

Yes

No

Meets Evidence Standards

Meets Evidence Standards with Reservations

Does Not Meet Evidence Standards


Standards account for overall and differential attrition rates l.jpg
Standards Account for Overall and Differential Attrition Rates

Sample maximum attrition thresholds for meeting WWC evidence standards


Caution 2 all qeds must demonstrate baseline between the treatment and control groups l.jpg
Caution 2: All QEDs must demonstrate baseline between the treatment and control groups

Randomization?

No

Yes

Attrition?

Low

Baseline Equivalence?

Yes

No

Meets Evidence Standards

Meets Evidence Standards with Reservations

Does Not Meet Evidence Standards


Baseline equivalence standard l.jpg
Baseline Equivalence Standard treatment and control groups

Equivalence must be demonstrated on the analytic sample


Other criteria l.jpg
Other Criteria treatment and control groups

  • Studies must measure impacts on relevant outcomes

  • Outcome measures must be reliable

  • Outcomes must not be over-aligned with the intervention

  • There is not a confound between the treatment condition and one or more other factors that also could be the cause of differences in outcomes between the treatment and comparison groups


Slide25 l.jpg
Questions & Answers treatment and control groupsPlease submit your questions on evidence eligibility requirements via the Webinar chat function now


Independent evaluation program requirement and guidance l.jpg
Independent Evaluation-- treatment and control groupsProgram Requirement and Guidance


Evaluation requirements l.jpg
Evaluation Requirements treatment and control groups

All i3 Grantees

MUST

MUST

MUST

Conduct an independent project evaluation

Cooperate with technical assistance provided by the Department or its contractors

Share the results of any evaluation broadly

Share the data sets for Validation and Scale-up evaluations

MUST

* Note: The quality of an applicant’s project evaluation is also a selection criterion.


Selection criterion c quality of project evaluation l.jpg
Selection Criterion C: Quality of Project Evaluation treatment and control groups

“The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes.”

Scale-up, Validation, and Development

“The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively.”

Scale-up, Validation, and Development

Understanding of Implementation and Intermediate Outcomes

Sufficient Funding to Carry Out Evaluation


Selection criterion c quality of project evaluation cont d l.jpg
Selection Criterion C: Quality of Project Evaluation (Cont’d)

“The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings.” Scale-up and Validation

“The extent to which the proposed evaluation will provide sufficient information about the key elements and approach of the project to facilitate further development, replication, or testing in other settings.” Development

Generates Information to Support Follow-on Scaling or Other Activities


Notes on selection criterion c quality of project evaluation l.jpg
Notes on Selection Criterion C. Quality of Project Evaluation

“The extent to which the methods of evaluation will include a well-designed experimental study, or if a well-designed experimental study of the project is not possible, the extent to which the methods of evaluation will include a well-designed quasi-experimental design.” Scale-up

“The extent to which the methods of evaluation will include a well-designed experimental study or a well-designed quasi-experimental study.” Validation

Design & Methodology for the Evaluation


Guidance on qualities of high quality evaluation plans l.jpg
Guidance on Qualities of EvaluationHigh-quality Evaluation Plans


Evaluation goals l.jpg
Evaluation Goals Evaluation

  • Aligned with i3 performance measures

  • Increase strength of evidence on the impact or promise of i3-supported interventions

  • i3 performance measures set the expectation for all i3 independent evaluations to provide high-quality implementation data and performance feedback

  • Other expectations vary by grant type


Evaluation goals scale up validation projects l.jpg
Evaluation Goals: EvaluationScale-up & Validation Projects

  • Estimate the impacts of the i3-supported intervention as implemented at scale

    • Expectation they will be “well-designed and well-implemented,” meaning designed, implemented and reported in a manner to meet WWC evidence standards

  • Provide information on key elements of the intervention for replication or testing in other settings

  • Reflect changes to the intervention or delivery model

  • Reflect the nature of sites served, implementation settings, and participants

  • Document costs/resource use


Evaluation goals development projects l.jpg
Evaluation Goals: EvaluationDevelopment Projects

  • Provide evidence of the promise of the practice, strategy, or program for improving student outcomes:

    • Using a research design that provides a comparison for the outcomes of the intervention group (e.g., a randomized controlled trial or strong matched comparison group design)

    • Using valid outcome measures (e.g., following the WWC standards)

  • Provide information on key elements of the intervention for further development, replication, or testing in other settings


Pointers based on the fy 2010 competition 1 of 6 l.jpg
Pointers Based on the FY 2010 Competition (1 of 6) Evaluation

  • Provide a fully developed plan in the application

    • Maximizes the ability to adequately address key questions

    • Minimizes the time spent at the beginning grant specifying details and negotiating revisions with ED

    • Key components of a high-quality evaluation plan

      • Logic model (What is the intervention? Who is it intended to serve? What outcomes is it expected to produce? How?)

      • Research questions (What do you want to learn? Who will the results pertain to?)

      • Proposed methods (What is the sampling frame? How will treatment and comparison groups be formed? What is the minimum detectable effect size? What data collection measures and methods will be used? How will the data be analyzed and reported?)

      • Coherent links among the above components


Pointers based on the fy 2010 competition 2 of 6 l.jpg
Pointers Based on the FY 2010 Competition (2 of 6) Evaluation

  • Provide a well-developed logic model to guide the evaluation plan:

    • Clear description of the i3-supported intervention/scaling mechanism

    • Key elements of the intervention in sufficient detail to guide replication

    • Target population for the innovation

    • Expected pathways through which the innovation is expected to affect intermediate and ultimate outcomes of interest

    • Expected mechanisms of change—e.g., shows how factors such as content, organization, duration, training, and other procedures are expected to lead to the intended outcomes


Pointers based on the fy 2010 competition 3 of 6 l.jpg
Pointers Based on the FY 2010 Competition (3 of 6) Evaluation

  • Clarity on key research questions in the evaluation plan

    • Creates shared expectations with partners and ED about what will be learned

  • Clarity on aspects of the logic model the evaluation will examine

  • Design (methods, sample, data collection, analysis) appropriate to address the key questions


Pointers based on the fy 2010 competition 4 of 6 l.jpg
Pointers Based on the FY 2010 Competition (4 of 6) Evaluation

  • Specify proposed methods

    • For questions about program effectiveness (i.e., causal inference questions):

      • Rely on experimental methods, when possible

        NOTE: Challenging to meet WWC evidence standards with quasi-experimental methods (as noted above)

      • Describe how treatment and control groups will be formed and monitored


Pointers based on the fy 2010 competition 5 of 6 l.jpg
Pointers Based on the FY 2010 Competition (5 of 6) Evaluation

  • Summarize data collection and analysis procedures

    • Describe data sources, samples, data collection procedures and timeline, and analysis methods for each research question

    • Ensure critical components of the logic model are represented in the data collection plan

      • NOTE: ensure measurement of implementation fidelity

      • NOTE: comparative studies should document the experiences of comparison participants


Pointers based on the fy 2010 competition 6 of 6 l.jpg
Pointers Based on the FY 2010 Competition (6 of 6) Evaluation

  • Ensure good match among logic model, research questions, proposed methods, and i3 goals:

    • Sample design should yield a sample representative of the relevant population group for the i3 project & support strong impact analyses (relevant to the intervention as implemented under i3)

    • Analysis plan should be appropriate for addressing the study questions and well-matched to the sample design

    • Plan for reporting the findings should be consistent with the evaluation goals and design


Questions answers please submit your questions on evaluation via the webinar chat function now l.jpg
Questions & Answers EvaluationPlease submit your questions on evaluation via the Webinar chat function now


Other important resources l.jpg
Other Important Resources Evaluation

  • Investing in Innovation Fund Website:

  • (http://www2.ed.gov/programs/innovation/index.html)

    • Notices of Final Revisions to Priorities, Requirements, and Selection Criteria

    • Application Packages for each competition (includes the respective Notice Inviting Applications)

    • Frequently Asked Questions

  • What Works Clearinghouse Website:

  • (http://ies.ed.gov/ncee/wwc)

    • Reference Resources, Procedures and Standards Handbook

    • Quick Review Protocol

All questions about i3 may be sent to [email protected]

Note: These slides are intended as guidance only. Please refer to the official Notices in the Federal Register.


ad