Michigan College Access NetworkDeveloping an Evaluation Plan Nathan Daun-Barnett, PhD April 13, 2011
Outline for Evaluation Presentation • Introduction of Programs • Understanding the Barriers to College Access • Developing a Theory of Change • Utilizing a Logic Model • Building a Data Plan
Levels of College Access Intervention State Policy MiCAP Know How 2 Go Social Marketing Community LCANs Promise Zones College Goal Sunday College Access Centers Institution School Practices MCAC College Outreach Individual/Family Preparation Navigating Process Affordability Finding Support
Academic Preparation • What do we know? • Better prepared students go to college at greater rates and are more likely to succeed • High demand for remediation limits chances for success • Approaches to the challenge • Rigorous High School Curriculum • Expanded Dual Enrollment Opportunities • ACT/MME • After School Tutoring & Mentoring
College Affordability • What do we know? • Low income students less likely to attend • Low SES also most sensitive to price in terms of choice • What we do? • Need-based Aid (Pell) • Merit-based Aid (former MI Promise) • Tuition Policies & Promise Zones • GI Bill and Veteran’s Benefits &TIP
Student Aspirations & Support • What do we Know? • Parents, peers and counselors influence aspirations and plans for college • Aspirations are high for all, but plans differ by race and SES • What do we Do? • MI Know How 2 Go • GEAR UP programs • College Positive Volunteers • NCAC College Advisors
Navigating the Process • What do we Know? • The process has grown increasingly complex • Efforts to facilitate the admissions process increases the likelihood of applying and receiving aid • What can we Do? • MiCAP Web Portal • College Goal Sunday/Tax Preparation • College Visits • College Success Centers
Develop a Theory of Change • Identify range of activities for your program • Link each activities to the barriers you believe are being addressed - including academic preparation, college affordability, social capital (aspirations and plans), and navigating the process
If these benefits to your programs are achieved, then certain changes in organizations, communities, or systems might be expected to occur If you accomplish your planned activities, then you will hopefully deliver the amount of product and/or service that you intended If you accomplish your planned activities to the extent you intended, then your programs will benefit in certain ways If you have access to them, then you can use them to accomplish your planned activities Certain resources are needed to operate your program Resources/ Inputs Activities Outputs Outcomes Impact Your Planned Work Your Intended Results
Reporting Outputs • This is an important set of formative measures and it is part of what we need to report to the federal government on the CACG • What are the anticipated outputs of your activities? • What are your expected goals for each of those metrics? (this serves as a point of comparison or baseline for the same measures one year later)
Summative Outcomes • The question we need to ask is “compared to what?” • Approach 1 - Compared to self at an earlier point. • Approach 2 - Compared to other similar students at same point.
Impact In order to measure impact, we must ask, Did our program improve the likelihood high school participants went to college?
How do we Measure Impact? Approach 1 - randomly assign students to the program Approach 2 - compare program students to a similar group of students (match sample) Approach 3 - statistically control for the factors accounting for the differences between groups Approach 4 - statistically control for selection bias
For more information, contact: Nathan Daun-Barnett firstname.lastname@example.org 716-645-1096
Developing a Data Plan Identifying Sources of Data • National Student Clearinghouse (enrollment, persistence, transfer, and completion) • School grade reports (transcripts) to capture grades, course completions, academic progress, attendance and behavior • ACT data includes both test scores for ACT, Workkeys, and MME in addition to student survey • MiCAP usage data (useful if part of intervention) • Student surveys to measure learning and behavior outcomes • Administrative records of program usage