Download
1 / 33

SHEEO Policy Conference August 9, 2012 - PowerPoint PPT Presentation


  • 101 Views
  • Uploaded on

1. SHEEO Policy Conference August 9, 2012. Sandy Baum, David Wright, Lee Holcombe, Kim Hunter-Reed. @ HCMStrat. Project background. Started in 2011 by HCM Strategists, with support from the Bill & Melinda Gates Foundation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' SHEEO Policy Conference August 9, 2012' - biana


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

1

SHEEO Policy Conference

August 9, 2012

Sandy Baum, David Wright, Lee Holcombe, Kim Hunter-Reed

@HCMStrat


Project background
Project background

  • Started in 2011 by HCM Strategists, with support from the Bill & Melinda Gates Foundation

  • Brought together researchers with different perspectives on higher education measurement

  • Asked research participants to think like policymakers and leaders

  • Asked policymakers and leaders to consider and respond to research


Key questions
Key questions

How should we account for different student populations when evaluating higher education outcomes?

  • What methods are available or in use for policymakers and leaders to account for variations in inputs?

  • What are the best ways to do it and what are the practical limitations?

  • Can research findings be translated into policy?


Project components

4

Project components

  • Seven white papers

  • Synopsis paper

  • Literature review

    Coming soon . . .

  • Policy briefs

  • Toolkit for states

  • Outreach--sharing findings, including more voices, continuing to solicit input


Some general lessons and conclusions
Some general lessons and conclusions

  • Input adjustments make a big difference

  • Policymakers should make an effort to adjust for inputs when institutional comparisons have stakes attached

  • Imperfect input adjustments are better than none

  • Need to be clear about outcomes


Readers will find specific guidance on
Readers will find specific guidance on

  • Important variables to include

  • Examples of adjustment models and statistical techniques

  • Use of survey for input adjustments

  • College learning assessments

  • Strategies for working with messy administrative data

  • Strengths and weaknesses of U.S. News input adjustments

  • Many potential unintended consequences


Where to find out more
Where to find out more . . .

Context for Success website:

www.hcmstrategists.com/contextforsuccess

Questions? Comments?


Advice for moving forward
Advice for Moving Forward

Know the purpose, audience and questions to be answered

  • Consumers: Judging quality and value

  • Institutions: Improving

  • Accountability: Success and cost-effectiveness


Advice for moving forward1
Advice for Moving Forward

Comparing outcomes: Focus on similar institutions

  • Differences in credentials

  • Goals of students

  • Role of transfer

    Selecting variables

  • Preparation

  • Demographics

  • Attitudes, goals, behaviors


Context for success white papers
Context for Success White Papers

Charles Clotfelter. "'Context for Success' Synopsis Paper."

Thomas Bailey and Di Xu. "Literature Review: Input Adjusted Graduation Rates and College Accountability."

Peter Riley Bahr. "Classifying Community Colleges Based on Students' Patterns of Usage."

Thomas Bailey. "Developing Input Adjusted Measures of Community College Performance."

Jesse M. Cunha and Darwin W. Miller. "Measuring Value-Added in Higher Education."


Context for success white papers cont
Context for Success White Papers (cont.)

Robert Kelchen and Douglas N. Harris. "Can 'Value Added' Methods Improve the Measurement of College Performance?"

Stephen M. Porter. "Using Student Learning as a Measure of Quality in Higher Education."

John Pryor and Sylvia Hurtado. "Using CIRP Student Level Data to Study Input Adjusted Degree Attainment."

David Wright, Matthew Murray, Bill Fox, Celeste Carruthers, Grant Thrall. "College Participation, Persistence, Graduation, and Labor Market Outcomes."


An Input-Adjusted Framework for Assessing the Effectiveness of Tennessee’s Higher Education InstitutionsContext for Success project

Celeste Carruthers,a William Fox,a Matthew Murray,a Grant Thrall,b

David L Wrightc

a University of Tennessee, Knoxville; b University of Florida;

c Tennessee Higher Education Commission


Background
Background of Tennessee’s Higher Education

  • Policy relevance:

    • Complete College TN Act ties HEI funding to student outcomes.

  • Assumption:

    • Funding incentives must take into account student characteristics that are likely to be important determinants of postsecondary outcomes.

Context for Success paper

SHEEO, 08-09-2012


Research questions
Research questions of Tennessee’s Higher Education

  • Purposes:

    • We statistically test whether some TN institutions were more effective than others at advancing the outcomes of 2002 entering first-time freshmen.

Context for Success paper

SHEEO, 08-09-2012


Research questions1
Research questions of Tennessee’s Higher Education

  • Purposes:

    • We go on to test whether accounting for student inputs (ACT aptitude, age, low-income home neighborhood, LifeMode marketing profile) changes the ranking of institutions.

      • Is model fit improved by supplementing or substituting our standard stock of administrative data with proprietary data on consumer tapestry profiles (called LifeModes) or with Census information?

Context for Success paper

SHEEO, 08-09-2012


Method
Method of Tennessee’s Higher Education

  • Cohort: 2002 first-time freshmen in TN public universities and community colleges

  • Design:

    • Multivariate OLS regression

    • Used fixed effects to isolate institutional effectiveness

Context for Success paper

SHEEO, 08-09-2012


Method1
Method of Tennessee’s Higher Education

  • Compared models with four different sets of predictor variables:

    • naïve model

    • standard state agency administrative data

    • Census neighborhood data

    • proprietary LifeModes profiles

  • Outcomes considered: student progression, transfer, graduation, and near-term labor market success

Context for Success paper

SHEEO, 08-09-2012


Selected findings
Selected findings of Tennessee’s Higher Education

  • Timely completion:

    • Females, higher entrance exam scores, and adult status were associated with more timely degree completion.

    • Low-income status delayed completion.

    • Additional semesters in college reduced earnings later on.

Context for Success paper

SHEEO, 08-09-2012


Selected findings1
Selected findings of Tennessee’s Higher Education

  • Earnings:

    • Females and higher entrance exam score were associated with higher earnings in the years immediately following college.

    • A significant black-white earnings gap favored white students.

Context for Success paper

SHEEO, 08-09-2012


Six-year graduation rates (bachelor’s) across 4-year schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

4-Year Institution

Context for Success paper

SHEEO, 08-09-2012


Four-year graduation rates (associate’s) across 2-year schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

2-Year Institution

Context for Success paper

SHEEO, 08-09-2012


Observations and open questions
Observations and open questions schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Institutions do matter: even after controlling for all observed student inputs, some institutions are significantly more effective than others.

    • Also see Cunha and Miller in this series.

  • Student inputs also matter: institutions’ apparent effectiveness changes when we account for student aptitude and background.

  • The gains from more data are not always high.

Context for Success paper

SHEEO, 08-09-2012


Observations and open questions1
Observations and open questions schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Why are some institutions more effective than others? Institutional effects are a black box.

  • Student goals and objectives may not align with traditionally tracked student outcomes.

    • Especially in community colleges!

  • Assessing and comparing student growth across fields and campuses is challenging but not insurmountable.

Context for Success paper

SHEEO, 08-09-2012


Conclusions
Conclusions schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Outcomes and institutional best practices should be included in performance-based funding. (p.6)

  • Incentivizing colleges for student progression and completion relative to expectations is an improvement over unadjusted rewards…

  • … but, be aware that outcomes will vary according to which students inputs are factored into expectations for colleges and universities. (p.32)

Context for Success paper

SHEEO, 08-09-2012


A project of the texas higher education coordinating board

Input Adjusted Output Measures schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

Presented By: Lee Holcombe, Director, Higher Education Policy Institute

Analysis: Jesse Cunha, of the Naval Postgraduate School and Trey Miller of the RAND Corporation

A Project of the Texas Higher Education Coordinating Board


Goals schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Develop and implement a methodology that generates quantitative metrics of institutional performance.

  • Come as close as possible to causal estimates of an institution’s impact on a particular outcome.

  • Demonstrate benefits of collecting additional data.

  • Develop a set of concrete and actionable policy recommendations for practitioners and policymakers.

Higher Education Policy Institute


Outcomes schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Persistence into 2nd year of college:

    • Attain 30 credit hours at any public or private four-year college in Texas by the end of the 6th academic year after graduating high school

  • Graduation:

    • Receive a Bachelor’s degree at any public or private four-year college in Texas by the end of the 6th academic year after graduating high school

  • Earnings:

    • Sum of 4 quarterly earnings in the 8th calendar year after high school graduation.

Higher Education Policy Institute


Data schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Higher education

    • Demographic variables, application and acceptance data, credits attempted, degree completion

  • K12 (Secondary)

    • Demographic variables, test scores, courses taken

  • Unemployment Insurance earnings

  • From College Board:

    • SAT scores and survey info

Higher Education Policy Institute


Sample schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Graduates of Texas public high schools in 5 cohorts (1998-2002)

  • Include those Texas public high school graduates who ever enrolled in a Texas public university (up to 2010).

  • 217,723 enrollees, 169,239 of whom had earnings greater than $2,000

Higher Education Policy Institute


Findings schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Large differences in unconditional outcomes across colleges diminish as each set of controls is added to the model.

  • Adding high school indicators and courses changes results much more than adding race and gender.

  • Different results across outcomes.

  • Measures are highly variable over time.

Higher Education Policy Institute


Policy Recommendations of Report schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Assessment of institutional quality should involve a broad set of outcomes representing multiple dimensions of institutional performance.

  • Researchers should work with state policymakers to develop models that reflect state priorities.

  • Do not assume that results are causal.

  • Create broad classifications of institutional performance as opposed to explicit rankings.

Higher Education Policy Institute


Policy Response schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

  • Elusive quest for causality.

  • Excludes many non-traditional students.

  • Consider improvements within-institution

  • Dig inside the higher education black box.

  • Provide higher levels of information and support.

  • Use of more nimble data systems.

  • Consider learning outcomes.

Higher Education Policy Institute


Where to find out more1
Where to find out more . . . schools, relative to one omitted institution, unadjusted (blue) or adjusted for student and home characteristics

COMING SOON

Context for Success website:

www.hcmstrategists.com/contextforsuccess

Questions? Comments?


ad