David Fairris Tarek Azzam

1. Evaluating Student Success: The Impact of a First-year Experience Program When Students Self-select David Fairris TarekAzzam This evaluation was funded by the United States Department of Education, Fund for the Improvement of Post Secondary Education (FIPSE) Program. The contents of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.

2. Introduction • A national dialogue has begun around the issue of improving college-going and college-completion rates. • First year persistence is a key indicator of college success, leading to graduation. • Many colleges and universities have turned to “first year experience” programs – such as first year seminars and learning communities – to enhance first-year retention and thus graduation. • How well are these programs working?

3. The Prevalence of First Year Experience Programs • First year experience programs are prevalent across much of higher education. • The National Resource Center for the First-Year Experience and Students in Transition found that 90% of reporting institutions possessed at least one freshman seminar on their campus (2012). • The John N. Gardner Institute for Excellence in Undergraduate Education found that 96% of four-year institutions possessed freshman seminars of one form or another (2010). • This same survey found that 91% of reporting institutions possessed a learning community of some form at their institution. • These are extraordinarily high participation numbers.

4. Student Participation and Self-Selection • Survey data do not offer precise measures of the extent to which freshman seminars or learning communities are populated by students who self-select into these programs. • According to the National Resource Center survey of 2012, 43% of institutions that possess freshman seminars require them of allincoming freshmen. • The Gardner Institute survey from 2009 yields an average student participation rate for seminars, among institutions that possess them, of 80%. • However, this masks great variation among institutions; for example, larger institutions (over 20,000 students) reported that less than half of their incoming freshman class participated in seminars. • The average student participation rate for learning communities in the Gardner Institute survey is much lower, at 33%.

5. Student Participation and Self-Selection • There is a high prevalence of first year experience programs across the higher education landscape • The numbers on participation rates suggest that the percentage of students enrolled in these programs through a voluntary process of self-selection is not insignificant. • Why does this matter for an evaluation of the effectiveness of these programs, and perhaps more generally for any academic or academic support program into which students self select?

6. The Freshman Seminar/Learning Community Structure Under Study One Year-Long Theme

7. Seminal Analysis Using a Quasi-Experimental Research Design (Fairris, Castro, & Son, 2010) • Enrollment is voluntary and students self select into the program • The analysis compares retention rates for treated self-selected students with non-self-selected students • Comparison of mean first-year retention rates • Comparison of conditional means using multiple regression analysis with controls for a host of student characteristics • Data were from the 2002 and 2003 freshman cohorts • Student characteristics such as high-school GPA, SAT scores, race, ethnicity, gender, and first-generation status were used as controls • Mean and conditional mean retention rates were statistically significantly higher for treated self-selected students

8. A Randomized Control Trial Research Design • A randomized control trial design was chosen as a follow-up exercise because: • It dealt with some of the selection bias concerns of previous evaluations. • Using a lottery system to assign students to the program was a logistical possibility. • It could be viewed as a more equitable way to distribute limited resources. • Previously it was first-come, first-served.

9. Cohort Design

10. Comparative Research Designs Not Self Selected (did not sign up for lottery) Self Selected (signed up for lottery) Treatment Students Control Students Other Students Quasi-Experimental Design Randomized Control Trial Design

11. Background Characteristics

12. Background Characteristics

13. Comparing RetentionTreatment vs. Control (ANOVA)

14. Comparing RetentionTreatment vs. Not Self Selected(ANCOVA) **p<.01

15. Comparing RetentionControl vs. Not Self Selected(ANCOVA) **p<.01

16. Program Impact Decomposition Equation Conventional Quasi-Experimental Result True Program Impact Selection Bias Not Self Selected Students Not Self Selected Students Treatment Students Control Students Treatment Students Control Students = + 92% - 87% = 5% 92% - 91% = 1% 91% - 87% = 4%

17. Conclusions • These results offer a “cautionary tale” regarding program evaluation in environments of student self selection. • Our results suggest that students who self-select into first year experience programs are a non-random draw from the larger freshman student population. They appear to be more motivated and committed to academic success. • When students self select, virtually all of the estimated program impact from a typical quasi-experimental, multiple regression research design is biased due to the positive selection of more motivated students.