NCATE Program Review Process

1. NCATE Program Review Process Margaret D. Crutchfield, Ph.D. September 2006 margie@ncate.org

2. A brief review…. • Unit = School, College, or Department of Education plus other entities on campus that prepare school personnel • Program = Specific Discipline Area

4. SPA = Specialized Professional Association

5. NCATE has standards for the following programs • Early Childhood Education • Educational Communications and Instructional Technology • Educational Computing and Technology • Educational Leadership • Elementary Education (Initial) • English as Second Language Education (Initial) • English Language Arts Education (Initial) • Foreign Languages Education (Initial) • Health Education (Initial) • Mathematics Education (Initial) • Middle School Education • Physical Education • Reading Education • School Library Media • School Psychology • Science Education (Initial) • Social Studies Education (Initial) • Special Education • Technology Education (Initial)

6. What about programs accredited by other accrediting bodies (e.g. ASHA, CACREP, etc.)? • What about programs for which NCATE does not have standards and are not accredited by another agency?

7. How Do Program Reports Fit into the Unit Accreditation Review? • Program recognition reports are important part of evidence for meeting NCATE Standard 1 • A unit can be accredited without receiving national recognition for all programs

8. Purpose of the Program Review • Determine whether or not the program has in place a limited number (6-8) of comprehensive assessments that demonstrate candidate mastery of the SPAstandards. • Candidate performance on these assessmentsis appropriate to demonstrate mastery. • Provide information for unit to use to respond to Unit Standard #1

9. NCATE Unit Standard 1: • Candidates preparing to work in schools as teachers or other school personnel know the content of their fields, demonstrate professional and pedagogical knowledge, skills, and dispositions and apply them so that students learn. Assessments indicate that candidates meet professional, state, and institutional standards.

10. Required Assessments • State licensure exam for program area (if available—otherwise another content based assessment) • Content Assessment • Assessment of Planning (e.g., unit plan) • Student teaching/internship assessment • Assessment of candidate impact on student learning or providing a supporting learning environment • A 6th assessment is required, type is optional

11. Required program assessments Content: • State licensure exam for program area (if available—otherwise another content based assessment) • Content Assessment Professional and Pedagogical Knowledge Skills and Dispositions: • Assessment of Planning (e.g., unit plan) • Student teaching/internship assessment

12. P – 12 Student Learning: • Assessment of candidate impact on student learning or providing a supporting learning environment Program is required to have at least one more assessment, can have a maximum of 8 to demonstrate candidate mastery of SPA standards

13. Assessment #1 - Licensure data • State test required for licensure in the content area • 80% of completers must meet or exceed the state pass score

14. Assessment #2: Another content-based assessment • Grades (although this can be tricky) • Comprehensive exam • Content-based portfolio assessment • Case studies; action research

15. Assessment # 3: Ability to plan Assessment of ability to plan instruction (classroom-based programs) • e.g., unit or lesson plan assignment Assessment of ability to fulfill specified professional roles (advanced programs) • e.g., needs assessment project

16. Assessment #4: Clinical Practice • Student teacher/internship/practicum evaluation What if it is a unit-wide (not discipline-specific) assessment?

17. Three effective strategies • SPA-specific addendum • SPA-specific rubrics • Coding assessment • AND including information in narrative about context

19. Assessment #5 : Impact on Student Learning Impact on Student Learning (Initial Teachers) • Teacher work sample or other classroom-based project Ability of Candidate to Provide Supportive Learning Environment (Other School Personnel) • Action research • Employer or other external surveys (ELCC)

20. Acceptable rubric for Unit Standard #1 expects candidates to • Assess student learning • Be able to use assessments in instruction • Develop meaningful learning experiences for students based on their developmental levels & prior experience.

21. Candidates can… • judge students’ prior learning, • plan and teach lessons, • assess student learning, • reflect on student learning, and • make adjustments to their teaching to improve learning.

22. Must have at least one more assessment (#6) but assessment type is optional unless SPA has SPA-specific requirements • SPA specific information is included in each program report form • Each program can have up to eight assessments

24. What is “an assessment”? • For the purposes of the program report, an assessment is an evaluated activity or requirement by which a program determines that specific outcomes or standards have been mastered by a candidate. • The assessment “package” consists of the assessment instrument, scoring guide, and candidate data derived from the assessment.

25. What isn’t an assessment • A syllabus (although it might be used as an assessment instrument for a course grade) • A list of program objectives or expected candidate outcomes • State or professional standards by which a program is designed/assessed

26. What might the assessment instrument look like? Examples • A sample of a comprehensive test • An evaluation form (student teaching, survey) • A course outline or outlines (if grades are used as the assessment) • A description of the assignment (unit plan, teacher work sample)

27. What Criteria Are Used to Evaluate Assessments? • Assessments are aligned to standards • Assessments reflect the rigor of standards • Assessments are free from bias • Scoring guides clearly identify levels of proficiency • Data provide evidence of candidate mastery of standards

28. What about the scoring guide? The scoring guide should: • Include “explicit statements of proficiencies candidates are expected to demonstrate in their responses.” • Be “constructed so that different levels of candidate proficiency are clearly distinguished.”

29. What isn’t a scoring guide A = Excellent B = Good C = Satisfactory etc. Journal = 10 pts Essay = 50 pts Attendance = 20 pts Final exam = 20 pts

31. More on Scoring Guides • The assessment/data are only as good as the scoring guide. • If possible, the scoring guide should be specific to the discipline and aligned to the standards. • Scoring guide should be correlated to and appropriate for the assessment.

32. Presenting Your Data • Data tables should be self-contained documents – readable and user-friendly • Shown the parts of the sum, not (just) the sum of the parts! • Data should be presented by categories used in the scoring guides

33. What Is Not Useful Data “ Candidate GPA in the math methods course for the past three years is 3.27.” “All of our candidates must score at the proficient level or above on this assessment in order to advance to student teaching.”

34. Characteristics of Well-Presented Data *Specify academic year or cohort group, e.g. “2002-3003”; “candidates admitted spring 2003” Note: Provide legends for abbreviations, acronyms, etc.

35. Disaggregate Data for • Different levels of a program (e.g. baccalaureate and Initial Masters) • Different sites at which a complete program is offered • Different cohorts or grading periods

36. How Much Data for Program Reports? • At least through reports submitted in spring 2007, one semester of data for five assessments • In fall 2007 and spring 2008, one year of data will be required • Fall 2008 and spring 2009, two years of data • Three years of data are optimal – not necessary to provide more

37. Program review decisions SPA reviewers will make one of the following decisions based on your program report: • The program is nationally recognized. • The program is nationally recognized with conditions. • Insufficient data • Insufficient alignment • Poor assessment, scoring guides, etc • 80% rule • The program is not nationally recognized; a new program report may be submitted.

38. When do you submit reports? • HIGHLY recommend submitting reports 12 months prior to site visit • February 1 or September 15 • Can submit up to 2 years prior to visit

39. What else has changed? • Not required to place key assessments at specific transition points • Do not submit examples of candidate work

40. Who to Contact at NCATE For questions related to overall process, submission requirements, etc. • Margie Crutchfield, margie@ncate.org • Jaye Bishop,jaye@ncate.org • Deidre Alves, deidre@ncate.org • Technical problems: Frank Huang, frank@ncate.org

41. Resources on NCATE web site • www.ncate.org, click on Institutions, click on Program Reviews • Program Report Forms, Guidelines, Instructions, Examples of Assessments, Examples of Report Sections • www.ncate.org, click on Program Reviewers • All training materials for program reviewers