1 / 14

S-STEM Program Evaluation

S-STEM Program Evaluation. S-STEM PI Meeting Arlington, VA October 2012. Program Evaluation Overview. NSF need for a comprehensive examination of implementation, effectiveness and impact of S-STEM Dual evaluation focus

doris
Download Presentation

S-STEM Program Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. S-STEM Program Evaluation S-STEM PI Meeting Arlington, VA October 2012

  2. Program Evaluation Overview • NSF need for a comprehensive examination of implementation, effectiveness and impact of S-STEM • Dual evaluation focus • Scholarship recipients and role of S-STEM in their educational opportunities and employment outcomes • Institutions and the student support they provide • Evaluation team: Abt Associates, SageFox, Abt SRBI

  3. Key Evaluation Features • Collaborative relations with NSF, projects, and other experts • Evaluation based in program theory • Involves 494 projects • Funded from 2006 to 2010 • Involve undergraduate students

  4. Challenges and Opportunities • Challenges • Tracking scholarship recipients • Obtaining cooperation and high response rates • Identifying an appropriate comparison • Opportunities • Collecting data representative of projects and program • Providing NSF with a strong understanding of investment • Contributing to understanding of student supports and success

  5. Study Components • Implementation study • Ways in which S-STEM projects recruit and retain students in STEM fields, allocate scholarship funds, and provide educational and support programming for scholarship recipients • Relational study • Associations between project characteristics and practices and recipient outcomes • Benchmarking comparison • Comparisons of recipients’ educational and academic support experiences to national trends • Quasi-experimental study • Use of Propensity Score Matching (PSM) to compare the educational and career outcomes of S-STEM scholarship recipients to a matched national sample

  6. Multiple Data Sources • S-STEM Monitoring System • Components and strategies employed by S-STEM projects • Projects’ Annual Reports • Descriptions of activities and implementation • Site visits to a purposive sample of S-STEM projects • In-depth information about project implementation in particular contexts • Survey data from S-STEM Principal Investigators • Operation and implementation of S-STEM awards • Survey data from S-STEM scholarship recipients • Information on educational experiences, and r educational and career outcomes • Beginning Postsecondary Students Longitudinal Study (BPS) survey • Extant data on outcomes from a matched comparison group

  7. Use of Monitoring System • Identify population • 5,478 at two-year institutions • 14,914 at four-year institutions • Identify supports and activities (2-year n=74, 4-year n=350) • Academic support services (96%, 94%) • Participation in meetings or conferences (86%, 63%) • Career counseling (76%, 70%) • Mentoring by faculty or professionals (73%, 94%)

  8. Site Visits to Selected Awards • Emerging themes • Building on an existing structure • Developing a community of scholars • Providing talent for the local workforce • Recruiting through an existing program • Providing early access to research • Recruiting to the university • Strategic alignment with other university initiatives • Mentoring/advising • Creating pathways

  9. PI Survey Topics • S-STEM activities • Recruitment and selection • Scholarship recipients • Recipient outcomes • Benefits and challenges

  10. Recipient Survey Topics • Education background • College experiences • Supports • Work • Financial aid • Future plans • Employment

  11. Critical Role of PIs and Recipients • Ensure high response rate on surveys • OMB threshold is 85% • Cooperation with requests from Abt Associates • Complete surveys • Encourage participation • Timeframe • NSF and OMB review required before fielding • Surveys in field Spring or Fall 2013

  12. Measure Progress toward Goals • Improved educational opportunities for students • Increased retention of students to degree achievement • Improved student support programs at institutions of higher education • Increased numbers of well-educated and skilled employees in technical area of national need

  13. Uses of Evaluation • Reporting progress and accomplishments to stakeholders • Congress, OMB, GAO, OSTP, NSF, the public • Accountability • Numbers of STEM graduates, time to degree, retention, success of project activities • Program improvement • Inform similar types of programs

  14. Questions alina_martinez@abtassoc.com

More Related