Five Year Self Study training

1. Five Year Self Study training Dr. Mary Hazzard, Assessment Fellow Dr. Jacque Caesar, Assessment Fellow October 23, 2013 (Notes taken at the training session and added here by SETM SAC Chair are in italic.)

2. OUTLINE • Learning Outcomes • Assessment at NU • Sections of Program Self-Study Report (PSSR) • Program Review • Purpose of Assessment • NU Assessment Plan • Possible Reflective Over-Arching Questions • Program Self-Study Committee (PSC) • Role of the School Assessment Committee (SAC) • Tips from a Program Lead • Rubrics • Components of Program Self-Study Report (PSSR) • Overarching Reflective Questions • Self-Study Plan • Section 1 – General Information • Section 2 – Relevancy of the Program • Section 3 – Currency of the Program

3. OUTLINE - Continued • Section 4 – Faculty Qualifications • Section 5 – Preparedness and Academic Support of Students • Section 6 – Student Achievement • Section 7 – Academic Success of Students • Section 8 – Program Vitality – Life Cycle Analysis • Section 9 – Adequacy of Resources • Section 10 – Other Information Needed for This Self-Study • Section 11 – Summary and Recommendations • Section 12 – External Reviewer(s) Report • External Review • Process for Contract for External Reviewer • Sample Contract 1 • Sample Contract 2 • AMS • Memorandum of Agreement (MOA) • Self-Study Plan: Summary • Review Process • Resources Provided by the Office of Institutional Research and Assessment (OIRA) • Summary

4. Learning Outcomes The program lead will be able to: • Identify overarching questions for the self study • Develop a self study plan with timeline • Identify members for a self study committee • Identify potential external reviewer(s) • Identify questions that could guide data collection and analysis for each section of the self study • Identify resources they could use as they complete the self study. • Complete a self study for their program.

5. Assessment at NU • Five Year Review • Based in Inquiry – what does the faculty need to know to improve the program • Evidence-based (relevant qualitative and quantitative evidence) decision making • Evidence of program quality • Curriculum and Learning Environment • Faculty • Evidence of program viability and sustainability • Information and technology resources • Facilities • Staff

6. Sections of Self-study (All in AMS) • Self Study Plan • Overarching Question(s) • General Program Information • Relevancy • Currency • Faculty qualifications • Preparedness and Academic support of students • Student achievement • Academic success of students • Program vitality • Adequacy of resources • Other • Summary and recommendations • External Review • Memorandum of Agreement (MOA)

7. Program review • Includes • Self-evaluation of program • Based in Inquiry – what does the faculty need to know to improve the program • Comprehensive analysis of program quality • Analysis of a wide variety of data about the program • Peer-evaluation by reviewers external to program, department, and university • Internal peer-evaluation • Used to: • Inform follow up planning • Inform budgeting process • Improve quality of program • Improve student learning • Update multi year assessment plan

8. Program review • The “program five-year review” refers to the program self-study activities, such as planning, coordinating, data collection, and data analysis, etc., performed by the program lead with his/her Program Self-Study Committee. • The scope of data to be included is data from the past five academic years or since the last five-year review was done. • The deliverables of a five-year review include: • A Five-Year Self-Study Plan • A Program Self- Study Report (PSSR) which also includes one report from each external reviewer • A Memorandum of Agreement (MOA)

9. Purpose of Assessment • Ensure evidence-based decision making • Provide accountability to the public and the profession • Adherence to adopted standards • Ensure students are learning • Ensure high quality programs • Encourage programmatic improvement • Ensure high quality graduates

10. NU Assessment Plan • To assure that we know what are students learn • Provide alignment across the mission and through the Institutional Learning Outcomes, Program Learning Outcomes, to the alignment of institutional resources • Activities designed to achieve the National University’s educational objectives • Transparent and collaborative process

12. Possible reflective Over-arching questions * Please choose one to three over-arching questions or compose your own over-arching questions relevant to your program and to be answered by the five-year review and in the PSSR: think of what need to be done to improve/maintain the academic program in the next five years? * Please include why the questions chosen are relevant to your program when you write the PSSR. * The chosen over-arching questions are to tie the whole review process together, not the sole questions to be addressed by the review. • Does the program meet the discipline criteria and/or national standards? • Does the program meet the needs of potential employers? • Is the curriculum current and relevant? • What do we need to do to meet the needs of tomorrow?

13. Questions continued • How can we improve retention and graduation rates? • How can we meet the needs of a diverse student body? • What do faculty need to improve pedagogy and improve curricular cohesion? • Do we need to refocus resources to better meet student needs? • Do we need to improve student support systems such as library, advising, technology, co-curricular activities to improve the academic success of students? • Do we need to refine the program learning outcomes and measures for assessment? • Are online courses designed and taught effectively? • Why has there been a decline in the number of students enrolled?

14. Sources of data as evidence for answering chosen over-arching questions and supporting other parts of the PSSR • Supporting data can be qualitative (from interviews or focus group discussions, etc.) • Supporting data can also be quantitative • Employer surveys, alumni surveys, adjunct surveys, student surveys (best done in their last classes before they become alumni and go elsewhere ) • Data analysis can compare data collected against external benchmarks or against data from other comparable university academic programs, etc.

15. Program self-study committee (PSC) • Full time and adjunct faculty who teach in program • Chair: Program Lead • Function: Advisory and committee members can draft some sections of the PSSR • Provide input and comments related to self-study plan and self-study • Members should be diverse as reflecting the class sections of the program: full-time, part-time, and adjunct instructors teaching online/on-site; teaching in San Diego/regions/online; teaching classes in various concentrations, etc. • Whether adjuncts get paid to be committee members is to be determined by the Dean’s office • Program lead should ask OIRA to grant access to committee members if committee members need full access to upload written documents into AMS • Formed: October and official line of completion is December 31 • Minimum: 3 members • Maximum: 5 to 6 members

16. Role of the SAC • Scott McClintock, Co-Chair SAC in COLS: • Tracking progress • Sharing best practices • Advising and mentoring when bottlenecks are encountered by program leads • Communicating between program leads and the University Academic Assessment Committee (UAAC), as necessary • Reviewing the PSSR in June 2013

17. Tips from a program Lead • Jacque, Co-Program Lead BAIS with Credential (self study completed 2012)

18. Rubrics • WASC rubrics for Program Learning Outcomes and Program Review • Five Year rubrics – Councils approved 2011 • Will be identified with each section.

19. WASC Rubric for Program Learning Outcomes

20. WASC Rubric for Program Review

21. Components of Self-Study • Overarching reflective questions • Develop 1-3 overarching questions to guide the self study

22. Self Study Plan • Complete the self study plan with target dates for completion and potential areas for review in each section. • Upload into AMS under the plan

23. Section 1 – General Information (Some of the content here is from the National University Catalog and CurriUNET) • How current is the catalog • How the program learning outcomes support the mission of the university, the school, and the department • Alignment of current program learning outcomes to institutional learning outcomes • Summarized history of changes to the program, including rationale for change • Description of professional development opportunities for students such as internships, publication/presentation, and the like • Summary of professional success of graduates: job placement/promotion data if available, notable contributions to the profession by program alumni.

24. Additional Thoughts • General information provides the context and sets the stage for the review • Internal context • External context • Might include any specializations or concentrations included • Goals of program (if applicable) (These are not the Program Learning Outcomes or PLOs) • Need for program (This may then be used to update the need in AMS PAR Standing requirements)

26. I. General Program Information • No rubric needed

27. Section 2 – Relevancy of the program • How well do program outcomes prepare graduates with the skills and knowledge expected by potential employers? • How well do program outcomes prepare graduates for professional exams? • How well do the program learning outcomes reflect the contemporary standards of the profession? • How relevant and aligned are the course learning outcomes to the program learning outcomes? • The library may be able to provide data for this section. • Interviewing employers and doing an alumni survey can be a source of such information.

30. II. Relevancy

31. Section 3 – Currency of the program • Do the program learning outcomes reflect current practice? • Does the program make use of current teaching strategies, curriculum structure, technology, delivery modalities, and the like? • Are the teaching materials provided to instructors kept current ? • Is each of the courses in the program maintained and updated regularly by one or more faculty who teach the course?

32. Additional Thoughts • In either relevancy or currency section: • How does the curriculum compare with curricula from other comparable institutions (the library may be able to supply the information for this question) • Is the curriculum map reflective of the progression throughout the program • Does the curriculum map reflect sufficient time for students to acquire the knowledge, skills, and values of the program outcomes

34. III. Currency

35. Section 4 – faculty qualifications • Are academic and professional qualifications of instructors specified for each course in the program ? • Do the instructors who teach the courses in the program have the appropriate academic and professional qualifications ? • Have any instructors been successful in obtaining funding for significant grants ? • Do they get adequate orientation and training, and if so, how do we know that ? • Are they engaged in continuous improvement of their teaching, for example by participation in best practice seminars and other sources? • Are they engaged in scholarly activities that enhance their knowledge of the subject matter they teach and/or their teaching strategies and skills ? • Do they keep current in their profession, and if so, how do we know that ? • Are their course outlines complete and appropriate for the course? • How effective are they as teachers? OIRA can supply summarized /aggregated student evaluation information here

36. Additional thoughts • How do faculty backgrounds, expertise, research, and other professional work contribute to the quality of the program • Proportion of faculty with terminal degrees • Distribution across ranks (data can be from OIRA) • Faculty service • Diversity of faculty (data can be from OIRA) • Awards and recognition of faculty • Record of professional practice • Institutions from which faculty earned degrees • Faculty specialties and how they align with program • If there were class sections in the regions, work with regional leads to analyze data by region as well.

42. IV. Quality of Faculty

43. Section 5 – Preparedness and academic support of students • Are there program-specific admission requirements that must be met, and if so how do we know that they are being met ? • Is there a means of enforcing prerequisites? If so, how do we know it is effective ? • Is there program-specific academic support for students in this program, and if so, what does it consist of? (example: organized faculty advisement)

44. Additional thoughts • Academic and career advising programs and resources • Tutoring. Supplemental instruction • Basic skill remediation • Support for connecting general learning requirements to discipline requirements • Orientation and transition programs • Support for engagement in campus community • Support for non-cognitive variables of success, e.g., psychological support for students or other flexibility for students who are parents with a lot of dependent children, etc. • Financial support, such as grants and scholarships

46. V. Preparedness Academic Success(Integration of institutional support and services)

47. Section 6 – Student achievement • How do we know the degree to which students are achieving the learning outcomes of the program ?

48. Additional thoughts • Annual results of direct and indirect assessment measures via reviewing and summarizing past Program Annual Reviews (PARs) in these five years • Ongoing efforts to close the loop by responding to assessment data and have the identified loops been closed? • Does curriculum offer sufficient breadth and depth of learning for the degree • Are the courses well sequenced and reliably available in sequence • How well does the curriculum align with the program outcomes • How well does the curriculum align with discipline standards, if appropriate • How should PAR be done in the future? • Here, this component of the PSSR emphasizes PLOs and assessment results

50. VI. Student AchievementSummary and Analysis of PLO’s