slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA PowerPoint Presentation
Download Presentation
Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA

Loading in 2 Seconds...

play fullscreen
1 / 39

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA - PowerPoint PPT Presentation

  • Uploaded on

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA. SFARN Conference Margaret Cahalan US Department of Education June 2009

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA' - Michelle

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Lessons Learned from the National Evaluation of Upward Bound and New Directions in light of 2008 HEOA

SFARN Conference

Margaret Cahalan

US Department of Education

June 2009

All tabulations and views reported in this presentation are the sole responsibility of the author and do not reflect any review, authorization or clearance by the US Department of Education

  • History Background—PART—Context-TRIO-HEOA
  • Transparent critical look at issues and re-analyses
  • Discuss Lessons Learned & New Models for Evaluation & Example from GEAR UP plans
clarification of what presentation is not intended to be
Clarification of What Presentation is Not Intended to Be
  • Critique of randomassignment-recognize power of method and hope will improve its application
  • Critique of Mathematica —disagree with some procedures and conclusions they reached but respect greatly the hard work and determination of completing this study
  • Act of Advocacy for the program —am acting as a researcher concerned with meeting research standards
upward bound 2008 overview
Upward Bound—2008 Overview
  • Year begun: 1965
  • Student Targeted: High School Students
  • Total funding: $328,528,379
  • Number of grants: 1017
  • Number served: 71,247
  • Average award: $323,036
  • Amount per person

served: $4,611

  • Avg. number funded

to serve per project: 70

study history
Study History

Controversial from beginning--Combined

Random assignment

probability sample weighted to produce nationally representative impact estimates

Second national evaluation-first one RTI in 1970s

Study was first random assignment study of UB: Begun in 1992—2004:

Four published impact reports 1996, 1999, 2004; Fourth follow up report unpublished

Fifth Follow up report—Published in January 2009

policy history
Policy History

Third Follow up--- reported few average overall effects; but large effects for students at-risk academically and with lower educational expectations

Newly devised OMB PART rating of “ineffective”; UB Initiative to serve at-risk--2002

Budget ---zero funding of federal pre-college programs in FY05 and FY06—dropped in FY07 and FY08

policy history cont
Policy History (cont)
  • UB 2006 Absolute Priority to serve 1/3 at-risk and 9th grade ;
  • New random assignment study to evaluate begun 2006
  • Congress blocked in 2007 and cancelled by ED in 2008
  • HEOA 2008
    • Mandates rigorous evaluations in TRIO
    • Prohibits over-recruitment for random assignment denial of services
    • Absolute Priority cancelled
personal involvement history
Personal Involvement History
  • PD for SSS and TS evaluations—survey director for third follow up—UB performance report analyses
  • Joined ED in late 2004 and study was under my team and working on Fourth Follow up
  • Concern when lead analyst from Fourth Follow up sent tables showing results sensitive to only one project
    • project had 26 percent of weight
    • seemingly large negative impacts
  • Review of fourth report concerns---2006-2008 began to look at data and consult with others concerning issues
basic assumptions of random assignment studies
Basic Assumptions of Random Assignment Studies

Sample representative of population to which wish to generalize

Treatment and control group are equivalent

Treatment and control group treated equally except for the treatment

Treatment and control group are mutually exclusive with regard to the treatment

examined 5 issues
Examined 5 Issues
  • Sample Design
  • Treatment Control Group Equivalency
  • Study attrition and non-response bias
  • Standardization of outcome measures
  • Control group service substitution and treatment drop-out
1 sample design issues
1. Sample Design Issues

Sample highly stratified—46 for 67 projects

Unequal weighting---One project has 26 percent, 3 projects 35, and 8 projects 50 percent of weight

Project level stratification—339—strata unequal within projects – average of 8 members per strata—target 3000 sample--

Basic Design Flaw--One project strata for largest strata—unable to estimate sampling variance

Serious representational issues with project 69—former 2-year representing 4-year

Treatment-control non-equivalency introduced by outlier 26 percent project

2 treatment control non equivalency
2. Treatment–Control Non-Equivalency

Sample well matched without project

Project 69 introduces bias into the overall sample in favor of the controls

Project 69 has large differences (examples)

Education expectations: 56 controls expect advanced degree—15 percent treatment

9th grade academics—8 percent controls at risk; 33 percent treatment at risk

Expected HS grad is 1997—60 treatment and 42 controls

3 lack of outcome standardization to expected high school graduation year ehsgy
3. Lack of Outcome Standardization to Expected High School Graduation Year (EHSGY)

Multi-grade study cohort (7 to 10 in 1992-93)

Randomization took place over 18 months

Small unbalances between treatment and control ---Control has larger percentage of older 10 grade students

Constructed standardized measures based on EHSGY

4 survey attrition and non response bias
4. Survey Attrition and Non-Response Bias

Concern in longitudinal studies

UB rates very high for follow ups but at 74 percent by end—control group 4-5 percent less response rate

Positive outcomes more likely to respond

Reports through fourth did not use Adm records---Use federal aid files to observe and impute

5 service participation and non participation issues
5. Service Participation and non-Participation Issues

Waiting List Drop-Outs --26 percent of treatment coded as not accepting the opportunity on waiting list file—kept in treatment sample

First Follow-up survey 18% non-participation in neither UB or UBMS in treatment group

Survey data--12-14 percent controls evidence of UB or UBMS participation

60 percent controls and 92 percent treatment group reported some pre-college supplemental service participation

dealing with issues approach of re analyses
Dealing with Issues --Approach of Re-Analyses
  • Sample design issues—present data weighted and unweighted
  • Bias in favor of Control Group
    • Present with and without project 69
    • Consider estimate for 74 percent not represented by 69 more robust
  • Standardize outcome measures by expected high school graduation
  • Positive survey response bias—used federal aid file records and NSC for some analyses
  • Substitution and Drop out—additional observational analyses
alternative analyses
Alternative Analyses

Experimental Analyses

Logistic regression--Intent to treat (ITT)—UB opportunity--original random assignment

Instrumental Variables Regression--Treatment on Treated (TOT)/ Complier Average Causal Effect (CACE)-UB/UBMS participation

Quasi-experimental—Observational-Instrumental variables regression

UB/UBMS compared to non-UB/non-UBMS service

Any service compared to no service

Selected subgroup (academic risk-and educational expectations)

baseline variables in model
Baseline Variables in Model

Used --sex, race/ethnicity, low income and first generation status, educational expectations, grade in 1992-93, grade on the student selection form, and whether the participant reported they had pre-college services prior to random assignment

Not use academic indicators in 9th grade

aspirations at base line
Aspirations at Base Line
  • 97 percent some postsecondary
    • 72 percent BA or higher
    • 25 percent below BA postsecondary
  • 3 percent no postsecondary
  • Enrollment
    • 68 percent in +18 months from EHSGY
    • 70 in +4 years from EHSGY
    • 75 by end of study period 7-9 years out
  • Postsecondary –any degree or certificate 35-47%
    • 47 percent any postsecondary degree or certificate by 7 to 9 years using only survey responders adjusted for non-response
    • 35 percent using complete sample with NSC data
  • BA degree—just over 20 percent by end of study period--+8
impact results
Impact Results

Significant and substantial positive ITT and TOT findings weighted and unweighted and with and without project 69 for:

Evidence of postsecondary entrance in +18 months and for +4 years

Application for financial aid in +18 months and for +4 years

Evidence of award of any postsecondary degree or credential by end of study period---driven by project 69 less than 4-year programs

awarded a ba in 6 years of ehsgy
Awarded a BA in +6 years of EHSGY
  • Weighted with 69 not sign. Unweighted sign.
  • For the 74 percent of sample not represented by project 69
    • 28 percent increase in BA award for ITT UB opportunity (13.3 increased to 17.0)
    • 50 percent increase in BA award for TOT UB participation analyses (14.1 to increased to 21.1)
sub group analyses
Sub-Group Analyses
  • Bottom 20 percent on academic indicators
    • Large positive significant effects for:
      • Postsecondary entrance
      • Application for financial aid
      • Award of any postsecondary degree
    • Not for BA degree –two few achieved to compare treatment and control
  • Top 80 percent on academic indicators

Moderate positive significant effects for:

      • Postsecondary entrance
      • Application for financial aid
      • Award of any postsecondary degree
      • For BA degree in +6
ub ubms participation compared with other non ub ubms services participation
UB/UBMS Participation Compared with Other non-UB/UBMS Services Participation

Quasi-experimental--Uses 2-stage instrumental variables regression—controls for selection bias not eliminate

Found statistically significant and substantive positive results for UB/UBMS participation for:

Evidence of postsecondary entrance +1 and +4

Application for financial aid +1 and +4

Award of BA in +6 unweighted overall and unweighted and weighted without project 69

  • Need to pay attention to sampling and non-sampling error issues even in random assignment studies—evaluation of evaluations
  • More listening to those being evaluated concerning what will work and not work—engagement of those evaluated-
  • Transparency—multiple groups analyzing data
  • ED use partnership model with projects for future UB studies
  • Gap between expectations and attainment—72 percent expect BA degree—just over 20 percent attain in +8 years
  • HEOA:2008 Contains limitation on certain types of random assignment studies that require recruitment of more students than would normally recruit for evaluation study purposes
  • How best to target resources and students to achieve goals and program improvement
  • How to focus program---How to serve at-risk students?
next steps trying with gear up
Next Steps—Trying with GEAR UP
  • Partnership model among stakeholders
  • Coordinated efforts across grantees
  • Utilized resources/leverage academic institutional research offices of grantees
  • Focus on program improvement rather than up or down
  • Open and transparent sharing
  • Build capacity for self evaluation and accountability
  • Utilization of standards for statistical research and program evaluation
gear up next generation
GEAR UP Next Generation
  • Concept of using data as feedback for program improvement
  • About to issue call for grantee partnerships to submit statements of interest in planning awards for rigorous studies—June 2009—
  • Small planning/proposal awards September 2009
  • Implementation awards 2010-2012—
contact information
Contact Information
  • 202-401-1679