Educational Realities

1. Alliances for Graduate Education and the ProfessoriateComparison Groups and Other issuesSeptember 18, 2008byCatherine M. Millett, Ph.D.Policy Evaluation & Research, ETS

2. Educational Realities The Circumstances: • Some may believe that graduate education is a luxury • Graduate degree holders could be viewed as an elite population – Among people 25 and over 1.3% have a Doctorate degree and 7% a Master’s degree • NSF receives 2.5% of the $166.5 billion federal budget for education in 2006 The Result: • All of us have to continually get out the word that graduate education is important and AGEP is important.

3. About AGEP • October 1998, NSF awarded 8 universities nearly $2.5 million each to significantly increase the number of African American, Hispanic and Native American students receiving SME degrees. • As of June 2008, there are 28 AGEP & SBE alliances • AGEP seeks to determine the college and university policies, practices, and support services that lead to increased numbers of minority SME undergraduates that then enter into graduate school and professorial careers.

4. Roosevelt’s Words of Wisdom • 2 cycles of the program have been completed. What do you expect? • Evolution of external perception • “This won’t work” • “AGEP isn’t working” • “While AGEP isn’t responsible for changes it is making a difference” • Now is the time to take stock of what has been accomplished and what needs to be accomplished.

5. Micro and Macro View of AGEP Who can best tell the Macro successes? Who can best tell the Micro successes?

6. Macro View - AGEP Alliance Models: FABRICATED DATA • Common goal: increase degree recipients 5

7. Individual Alliance Reports • Showcase the Alliance’s accomplishments – Veronica told us to “Tell Your Story” • What is your AGEP student experience? • Think about the various audiences for your reports – NSF, public policy makers, university administrators, other faculty, the public, prospective and current graduate students. • What are we paying for and why? (e.g. trips to conferences, faculty recruiting trips?) • Provide information to NSF and other evaluators that may not be in AGEP wide report: • Students’ presentations and awards • Press coverage • NSF employees participation in Alliance events • How you are leveraging AGEP funding

8. Figure 2 – Hierarchy of Study Designs for Evaluating the Effectiveness of a STEM Education Intervention, by Expected Distribution of Study Type EXPERIMENTAL: Such as Well-designed Randomized Controlled Trials QUASI EXPERIMENTAL: Such as Well-matched Comparison-Group Studies OTHER DESIGNS: Such as Pre-Post Studies, and Comparison-Group Studies without careful matching U.S. Department of Education. (May 2007) Report of the Academic Competitiveness Council Source: Coalition for Evidence-Based Policy, 2006

9. Putting Your Date in Context - Graduate Education Sources • Graduate Record Examinations – Guide to the Use of Scores 2207-2008. • US Department of Education (NCES 2007-162). The Path Through Graduate School; A Longitudinal Examination 10 Years After Bachelors’ Degree • US Department of Education (NCES2006-185). Student Financing of Graduate and First-Professional Education, 2003-2004: Profiles of Students in Selected Degree Programs and Part-Time Students • Survey of Earned Doctorates: 2006 Doctorate Recipients from United States Universities: Summary Report • http://nces.ed.gov/das/library/nedrc_tables.asp • Professional societies may have data.

10. Variations in Defining AGEP Population All STEM Students URM STEM Students AGEP Students

11. Comparison Groups – The Goal Fill in the sentence: AGEP could be making a difference in…. Answer the question: Is your program making a difference? Goal of using comparison groups – provide credible evidence that your program could be making a difference The Ideal Members: • Graduate students who have not participated in AGEP • Graduate students who are similar to AGEP participants • Have characteristics that the research base demonstrates are important relative to the intended outcomes of the program. Random Assignment to AGEP Treatments • Do you have an adequate number of students to do random assignment? • Do faculty and administrators support random assignment (e.g. in admissions?) • Could you randomly assign students to other experiences in the AGEP program (e.g. half of students receive full conference funding versus the other half who get 50% of conference support).

12. Other issues to consider Naturally Found Groups • Applicants accepted to your graduate program but not supported by AGEP. • Graduate students from pre-AGEP years • Graduate students who participate in other educational programs from other agencies (e.g. NIH, Energy). • Students in graduate schools who are not in the AGEP Alliances Possibilities: • Students who attend summer orientation versus those who do not. • Students who attend semester long series on writing research papers compared to those who do not. • Students who take a particular academic class (.e.g. statistics) versus those who do not.

13. Practical Issues Securing Human Subjects Approval Accessing data Planning for staff time and ability Incentives Response rates Cost – if you have an evaluation budget remember to report how it is being spent and if you need more talk to your program officer. All they can say is no. Confidentiality in reporting data

14. Get the A for Effort!