Assessment and Evaluation of CAREER Educational Components

1. Assessment and Evaluation of CAREER Educational Components monica.devanas@rutgers.edu Center for Teaching Advancement and Assessment Research

2. The Project Description … should include: • a description of the proposed research project, including preliminary supporting data where appropriate, specific objectives, methods and procedures to be used, and expected significance of the results; • a description of the proposed educational activities, including plans to evaluate their impact on students and other participants; • a description of how the research and educational activities are integrated with one another; and • results of prior NSF support, if applicable. http://www.nsf.gov/pubs/2014/nsf14532/nsf14532.htm

3. Successful applicants will propose creative, effective, integrated research and education plans, and indicate how they will assess these components. While excellence in both education and research is expected, activity of an intensity that leads to an unreasonable workload is not. The research and educational activities do not need to be addressed separately if the relationship between the two is such that the presentation of the integrated project is better served by interspersing the two throughout the Project Description. http://www.nsf.gov/pubs/2014/nsf14532/nsf14532.htm

4. Education Program Goals • designing innovative courses or curricula • supporting teacher preparation and enhancement • conducting outreach and mentoring activities to enhance scientific literacy or involve students from groups that have been traditionally underrepresented in science • researching students' learning and conceptual development in the discipline • incorporating research activities into undergraduate courses • providing mentored international research experiences for U.S. students • linking education activities to industrial, international, or cross-disciplinary work • implementing innovative methods for evaluation and assessment.

5. Educational Program Review Education Program Goals • 3-5 overarching goals • Integrate research areas into curriculum • Increase applicant pool to your program • Connect with international scholars • Summer internships for high school students

6. Educational Program Review Education Program Goal #1 Activities • Integrate research areas into curriculum • New course • Course portfolio, syllabus, student projects • Pre/post tests of knowledge, skills and attitudes • Surveys of students • Focus groups • Feedback from colleagues in department • Enrollment increase over time • Student success in subsequent courses, career options

7. Educational Program Review Education Program Goal #2 Activities • Increase applicant pool to your program • Enrollments from past years compared to post award • Demographics of applicants • Underrepresented • SAT’s, GPA’s, GRE’s • Student projects • Surveys of students, exit surveys, • Feedback from faculty

8. Educational Program Review Education Program Goal #3 Activities • Connect with international scholars • Collaborative projects • Joint publications, symposia • Inter-institutional visits, graduate student exchange • Interviews, surveys, student work • Retention, success in advanced work • International Post-graduate work • Measureable outcomes at both sites

9. Educational Program Review Education Program Goals • 3-5 overarching goals • Program Activities • Assessment and Evaluation

10. Cycle of Assessment/Evaluation NSF Handbook for Project Evaluation, page 4

11. Cycle of Assessment/Evaluation New course Pre/post test Survey students Faculty Feedback Assessment Committee Update field State–of--Art Content in class and labs NSF Handbook for Project Evaluation, page 4

12. Types of Evaluation NSF Handbook for Project Evaluation, page 8

13. Types of Evaluation Formative NSF Handbook for Project Evaluation, page 8

14. Types of Evaluation Formative As promised For improvement Assessment Committee Assessment Committee Assessment Committee External Reviewer External Reviewer NSF Handbook for Project Evaluation, page 8

15. Measureable Outcomes NSF Handbook for Project Evaluation, page 18

16. Measureable Outcomes New Course NSF Handbook for Project Evaluation, page 18

17. Kinds of Assessments Direct Assessments Indirect Assessments Qualitative Assessments Quantitative Assessments

18. Kinds of Assessments Direct Assessments Indirect Assessments Departmental survey Exit interviews Alumni survey Employer survey Focus groups Job placement statistics Graduation and retention rates Percentage of students who study abroad Pre and posttests Course-embedded assessment ( homework assignment; essays, locally developed tests) Comprehensive exams National Major Field Achievement Tests GRE subject exams Certification exams, licensure exams Senior thesis or major project Portfolio evaluation Case studies Reflective journals Capstone projects Internship and clinical evaluation External examiners/peer review Grading with criteria or rubrics

19. Kinds of Assessments Quantitative Assessments Qualitative Assessments Observations Interviews Focus Groups Non-statistical Surveys/Questionnaires Pre/post Tests Existing Databases Statistical Analysis

20. Kinds of Assessments Direct Assessments Pre/post test, student work Indirect Assessments Exit Interview/survey, focus group Qualitative Assessments Interviews, observations Anecdotal examples Quantitative Assessments Statistical analysis of surveys Student demographics

21. Assessment Oversight • Internal Assessment Committee • PI • Departmental colleagues • Other colleagues not affiliated with project • Meet regularly to review process and formative assessments • 1-2/semester • External Reviewer • Assessment “expert” • Not affiliated with institution, or project • Consult on Assessment Plan for grant • Examine implementation process during grant • Final Summative Report • 1/year

22. Kinds of Assessments Direct Assessments Pre/post test, student work Indirect Assessments Exit Interview/survey, focus group Qualitative Assessments Interviews, observations Anecdotal examples Quantitative Assessments Statistical analysis of surveys Student demographics

23. Educational Program Review Education Program Goal #1 Activities • Integrate research areas into curriculum • New course • Course portfolio, syllabus, student projects • Pre/post tests of knowledge, skills and attitudes • Surveys of students • Focus groups • Feedback from colleagues in department • Enrollment increase over time • Student success in subsequent courses, career options

24. Educational Program Review Education Program Goal #2 Activities • Increase applicant pool to your program • Enrollments from past years compared to post award • Demographics of applicants • Underrepresented • SAT’s, GPA’s, GRE’s • Student projects • Surveys of students, exit surveys, • Feedback from faculty

25. Educational Program Review Education Program Goal #3 Activities • Connect with international scholars • Collaborative projects • Joint publications, symposia • Inter-institutional visits, graduate student exchange • Interviews, surveys, student work • Retention, success in advanced work • International Post-graduate work • Measureable outcomes at both sites

26. Educational Program Review Education Program Goal #4 Activities • Summer internships for high school students • Recruitment activities, website • Connections and programs for teachers • Engagement and feedback • Profile of participants, women, underrepresented groups • Mentoring and advising activities by faculty and grad students • Study skill support, pre/post tests of knowledge, skills and attitudes • Student projects, essays, surveys, exit interview • Follow-up with students, college selection • Subsequent increase in interest, enrollment, faculty partners

27. Best Practices for Assessment Have an assessment plan Balance Direct/Indirect Quantitative/Qualitative Assessment committee External Reviewer ? Collect everything you can Talk to every one you can Present/publish research & education activities