Lessons Learned
1 / 39

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA - PowerPoint PPT Presentation

  • Updated On :

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA. SFARN Conference Margaret Cahalan US Department of Education June 2009

Related searches for Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA' - Michelle

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA

SFARN Conference

Margaret Cahalan

US Department of Education

June 2009

All tabulations and views reported in this presentation are the sole responsibility of the author and do not reflect any review, authorization or clearance by the US Department of Education

Topics l.jpg

  • History Background—PART—Context-TRIO-HEOA

  • Transparent critical look at issues and re-analyses

  • Discuss Lessons Learned & New Models for Evaluation & Example from GEAR UP plans

Clarification of what presentation is not intended to be l.jpg
Clarification of What Presentation is Not Intended to Be

  • Critique of randomassignment-recognize power of method and hope will improve its application

  • Critique of Mathematica —disagree with some procedures and conclusions they reached but respect greatly the hard work and determination of completing this study

  • Act of Advocacy for the program —am acting as a researcher concerned with meeting research standards

Upward bound 2008 overview l.jpg
Upward Bound—2008 Overview

  • Year begun: 1965

  • Student Targeted: High School Students

  • Total funding: $328,528,379

  • Number of grants: 1017

  • Number served: 71,247

  • Average award: $323,036

  • Amount per person

    served: $4,611

  • Avg. number funded

    to serve per project: 70

Study history l.jpg
Study History

Controversial from beginning--Combined

Random assignment

probability sample weighted to produce nationally representative impact estimates

Second national evaluation-first one RTI in 1970s

Study was first random assignment study of UB: Begun in 1992—2004:

Four published impact reports 1996, 1999, 2004; Fourth follow up report unpublished

Fifth Follow up report—Published in January 2009

Policy history l.jpg
Policy History

Third Follow up--- reported few average overall effects; but large effects for students at-risk academically and with lower educational expectations

Newly devised OMB PART rating of “ineffective”; UB Initiative to serve at-risk--2002

Budget ---zero funding of federal pre-college programs in FY05 and FY06—dropped in FY07 and FY08

Policy history cont l.jpg
Policy History (cont)

  • UB 2006 Absolute Priority to serve 1/3 at-risk and 9th grade ;

  • New random assignment study to evaluate begun 2006

  • Congress blocked in 2007 and cancelled by ED in 2008

  • HEOA 2008

    • Mandates rigorous evaluations in TRIO

    • Prohibits over-recruitment for random assignment denial of services

    • Absolute Priority cancelled

Personal involvement history l.jpg
Personal Involvement History

  • PD for SSS and TS evaluations—survey director for third follow up—UB performance report analyses

  • Joined ED in late 2004 and study was under my team and working on Fourth Follow up

  • Concern when lead analyst from Fourth Follow up sent tables showing results sensitive to only one project

    • project had 26 percent of weight

    • seemingly large negative impacts

  • Review of fourth report concerns---2006-2008 began to look at data and consult with others concerning issues

Basic assumptions of random assignment studies l.jpg
Basic Assumptions of Random Assignment Studies

Sample representative of population to which wish to generalize

Treatment and control group are equivalent

Treatment and control group treated equally except for the treatment

Treatment and control group are mutually exclusive with regard to the treatment

Examined 5 issues l.jpg
Examined 5 Issues

  • Sample Design

  • Treatment Control Group Equivalency

  • Study attrition and non-response bias

  • Standardization of outcome measures

  • Control group service substitution and treatment drop-out

1 sample design issues l.jpg
1. Sample Design Issues

Sample highly stratified—46 for 67 projects

Unequal weighting---One project has 26 percent, 3 projects 35, and 8 projects 50 percent of weight

Project level stratification—339—strata unequal within projects – average of 8 members per strata—target 3000 sample--

Basic Design Flaw--One project strata for largest strata—unable to estimate sampling variance

Serious representational issues with project 69—former 2-year representing 4-year

Treatment-control non-equivalency introduced by outlier 26 percent project

2 treatment control non equivalency l.jpg
2. Treatment–Control Non-Equivalency

Sample well matched without project

Project 69 introduces bias into the overall sample in favor of the controls

Project 69 has large differences (examples)

Education expectations: 56 controls expect advanced degree—15 percent treatment

9th grade academics—8 percent controls at risk; 33 percent treatment at risk

Expected HS grad is 1997—60 treatment and 42 controls

3 lack of outcome standardization to expected high school graduation year ehsgy l.jpg
3. Lack of Outcome Standardization to Expected High School Graduation Year (EHSGY)

Multi-grade study cohort (7 to 10 in 1992-93)

Randomization took place over 18 months

Small unbalances between treatment and control ---Control has larger percentage of older 10 grade students

Constructed standardized measures based on EHSGY

4 survey attrition and non response bias l.jpg
4. Survey Attrition and Non-Response Bias Graduation Year (EHSGY)

Concern in longitudinal studies

UB rates very high for follow ups but at 74 percent by end—control group 4-5 percent less response rate

Positive outcomes more likely to respond

Reports through fourth did not use Adm records---Use federal aid files to observe and impute

5 service participation and non participation issues l.jpg
5. Service Participation and non-Participation Issues found on aid files 1994-2004

Waiting List Drop-Outs --26 percent of treatment coded as not accepting the opportunity on waiting list file—kept in treatment sample

First Follow-up survey 18% non-participation in neither UB or UBMS in treatment group

Survey data--12-14 percent controls evidence of UB or UBMS participation

60 percent controls and 92 percent treatment group reported some pre-college supplemental service participation

Dealing with issues approach of re analyses l.jpg
Dealing with Issues --Approach of Re-Analyses found on aid files 1994-2004

  • Sample design issues—present data weighted and unweighted

  • Bias in favor of Control Group

    • Present with and without project 69

    • Consider estimate for 74 percent not represented by 69 more robust

  • Standardize outcome measures by expected high school graduation

  • Positive survey response bias—used federal aid file records and NSC for some analyses

  • Substitution and Drop out—additional observational analyses

Alternative analyses l.jpg
Alternative Analyses found on aid files 1994-2004

Experimental Analyses

Logistic regression--Intent to treat (ITT)—UB opportunity--original random assignment

Instrumental Variables Regression--Treatment on Treated (TOT)/ Complier Average Causal Effect (CACE)-UB/UBMS participation

Quasi-experimental—Observational-Instrumental variables regression

UB/UBMS compared to non-UB/non-UBMS service

Any service compared to no service

Selected subgroup (academic risk-and educational expectations)

Baseline variables in model l.jpg
Baseline Variables in Model found on aid files 1994-2004

Used --sex, race/ethnicity, low income and first generation status, educational expectations, grade in 1992-93, grade on the student selection form, and whether the participant reported they had pre-college services prior to random assignment

Not use academic indicators in 9th grade

Aspirations at base line l.jpg
Aspirations at Base Line found on aid files 1994-2004

  • 97 percent some postsecondary

    • 72 percent BA or higher

    • 25 percent below BA postsecondary

  • 3 percent no postsecondary

Descriptive l.jpg
Descriptive found on aid files 1994-2004

  • Enrollment

    • 68 percent in +18 months from EHSGY

    • 70 in +4 years from EHSGY

    • 75 by end of study period 7-9 years out

  • Postsecondary –any degree or certificate 35-47%

    • 47 percent any postsecondary degree or certificate by 7 to 9 years using only survey responders adjusted for non-response

    • 35 percent using complete sample with NSC data

  • BA degree—just over 20 percent by end of study period--+8

Impact results l.jpg
Impact Results found on aid files 1994-2004

Significant and substantial positive ITT and TOT findings weighted and unweighted and with and without project 69 for:

Evidence of postsecondary entrance in +18 months and for +4 years

Application for financial aid in +18 months and for +4 years

Evidence of award of any postsecondary degree or credential by end of study period---driven by project 69 less than 4-year programs

Applicant on aid file l.jpg
Applicant on Aid File found on aid files 1994-2004

Awarded a ba in 6 years of ehsgy l.jpg
Awarded a BA in +6 years of EHSGY found on aid files 1994-2004

  • Weighted with 69 not sign. Unweighted sign.

  • For the 74 percent of sample not represented by project 69

    • 28 percent increase in BA award for ITT UB opportunity (13.3 increased to 17.0)

    • 50 percent increase in BA award for TOT UB participation analyses (14.1 to increased to 21.1)

Sub group analyses l.jpg
Sub-Group Analyses found on aid files 1994-2004

  • Bottom 20 percent on academic indicators

    • Large positive significant effects for:

      • Postsecondary entrance

      • Application for financial aid

      • Award of any postsecondary degree

    • Not for BA degree –two few achieved to compare treatment and control

  • Top 80 percent on academic indicators

    Moderate positive significant effects for:

    • Postsecondary entrance

    • Application for financial aid

    • Award of any postsecondary degree

    • For BA degree in +6

Ub ubms participation compared with other non ub ubms services participation l.jpg
UB/UBMS Participation Compared with Other non-UB/UBMS Services Participation

Quasi-experimental--Uses 2-stage instrumental variables regression—controls for selection bias not eliminate

Found statistically significant and substantive positive results for UB/UBMS participation for:

Evidence of postsecondary entrance +1 and +4

Application for financial aid +1 and +4

Award of BA in +6 unweighted overall and unweighted and weighted without project 69

Lessons l.jpg
Lessons Services Participation

  • Need to pay attention to sampling and non-sampling error issues even in random assignment studies—evaluation of evaluations

  • More listening to those being evaluated concerning what will work and not work—engagement of those evaluated-

  • Transparency—multiple groups analyzing data

  • ED use partnership model with projects for future UB studies

  • Gap between expectations and attainment—72 percent expect BA degree—just over 20 percent attain in +8 years

  • HEOA:2008 Contains limitation on certain types of random assignment studies that require recruitment of more students than would normally recruit for evaluation study purposes

  • How best to target resources and students to achieve goals and program improvement

  • How to focus program---How to serve at-risk students?

Next steps trying with gear up l.jpg
Next Steps—Trying with GEAR UP Services Participation

  • Partnership model among stakeholders

  • Coordinated efforts across grantees

  • Utilized resources/leverage academic institutional research offices of grantees

  • Focus on program improvement rather than up or down

  • Open and transparent sharing

  • Build capacity for self evaluation and accountability

  • Utilization of standards for statistical research and program evaluation

Gear up next generation l.jpg
GEAR UP Next Generation Services Participation

  • Concept of using data as feedback for program improvement

  • About to issue call for grantee partnerships to submit statements of interest in planning awards for rigorous studies—June 2009—

  • Small planning/proposal awards September 2009

  • Implementation awards 2010-2012—

Contact information l.jpg
Contact Information Services Participation