Using Employer Data to Assess Program Effectiveness

1. Using Employer Data to Assess Program Effectiveness Hillary Merk Randy Hetherington Bruce Weitzel James Carroll Jacqueline Waggoner

2. CAEP Definitions of Completer • Two definitions of completer- Standard 1 and 4 “A term to embrace candidates exiting from… preparation programs.” (Standard 1 glossary) Program completers who have been out of the program at least 6 months.

3. Objectives • To share how we tell if our programs have adequately prepared our completers for today’s classrooms through a process of tracking them into their early careers. • Explore the sources of data that can be used for continuous program improvement. • Discuss how employer satisfaction data, specifically principal interviews, can be used as part of the continuous program improvement cycle.

4. Program Improvement Data • EPPs must demonstrate measures are reliable and valid. • There is a need to gather a minimum of three cycles of informative data prior to an accreditation review. • It takes time to implement curricular changes in response to identified areas for improvement. • Curricular changes require continuous assessment of completer impact in P-12 schools.

5. Multiple Measures Approach • Develop an array of measures: • Single measures may not portray a complete picture of the impact of our EPP. • Cumulatively they provide convincing evidence. • Collectively they are a reflection of the impact of our programs. • They demonstrate program efficacy as it links to program knowledge and skills (curriculum).

6. Standard 4 • Impact on Student Learning • Indicators of Teaching Effectiveness • Satisfaction of Completers • Satisfaction of Employers

7. Standard 4: Impact on Student LearningPre/Post Assessment on a Unit of Instruction • Pre/Post assessments use to measure specific, teacher-selected, instructional units. • Unit assessments were designed around matched pre/post assessments of students. • Activity mimicked work that the teachers had done twice while they were candidates in their preparation program.

8. Standard 4: Impact on Student LearningPre/Post Assessment on a Unit of Instruction • Completers report on an Excel spreadsheet pre/post scores of each P-12 student along with unit goals, and demographic categories such as gender, ethnicity, and identified learning needs. • Learning gains computed for each P-12 student as the percent correct difference between pre-and post assessments. • Completers disaggregate and analyze data.

9. Standard 4: Indicators of Teaching EffectivenessClassroom Observations • A sample of volunteer teachers who had completed the analysis of a unit of study were observed teaching in their classrooms by a clinical practice professor. • Rubric developed to align with the mid-term and final evaluation used in their student teaching. • The observers used open-ended notes to each of the rubric elements to avoid teachers feeling that the observation was evaluative. • Principals of those who volunteered to be observed were also interviewed.

10. Standard 4: Satisfaction of Completers • Data analyzed from: • State survey of new teachers • New Teacher Support Group for recently hired candidates • Educational Leadership Network • Teacher Leadership Network • Focus groups of completers

11. Standard 4: Satisfaction of EmployersEmployer Interviews • Interviews of principals of newly hired EPP graduates (of both the four-year undergraduate and the 10-month MAT programs) • Associate Dean has face-to-face interviews with Principals and Vice Principals • Principals or Vice Principals fill out short survey • The Principals interviewed had completers in their building who submitted the Pre/post test and agreed to a classroom observation

12. Standard 4: Satisfaction of EmployersEmployer Interviews • Data collected using a Likert scale survey and through face-to-face interviews administered (recorded) every two years. • Transcripts from employer interviews received first cycle descriptive coding and sub-coding followed by second cycle axial coding (Saldaña, 2013). • The resulting categories were compared across the four question prompts and repeated iterations through the raw data revealing four specific themes for discussion: Training; Classroom Culture, Relationships and Teacher Identity. • Each theme contains evidence of effective program components as well as recommendations for potential improvements.

13. Training • The training theme encompassed the learning environment and skill acquisition of completers. • Three categories emerged within the theme which are illustrative of employer (principal) feedback: Assessment, Pedagogy and Classroom Management. • Each category represented areas of strength exhibited by completers and areas for consideration in program improvement.

14. Training • In terms of assessment, completers were praised for utilizing a variety of formative methods, and others encouraged to learn more about analyzing data, and using data to modify or adapt pedagogy. • P3 “…they require more knowledge and practice with different kinds of [standardized] assessments. • P20 “…they’re taking data, looking at it, going back and changing their practice based on the data.” • P29 “…she had a full range of learners including struggling learners and ESL….she had SPED students and she was impressively able to touch all of these kids, and see growth in all of them.”

15. Training • In the area of pedagogy, the majority of candidates were praised for attention to issues of diversity and equity and encouraged to further develop methods of differentiation of instruction and ways to best integrate technology to improve student engagement. • P15 “…need to look for ways to really infuse technology into classrooms, not just to replace pen and paper but to use as a tool in everyday life.” • P14 “…she definitely differentiates between learning styles. You can tell she came out of a strong program.” • P12 “…you can really see how she scaffolds and differentiates [instruction] for her kids.”

16. Training • Almost universal was principal acknowledgment of completers establishing a caring and culturally responsive learning environment. • P13 “This is one of the most diverse schools in XXXX County in terms of race, politics, and socio-economic status and she handles it beautifully.” • Potential for growth using technology to better organize student learning and as a result improve student on-task behavior. • P23 “Personalized learning and using technology to provide students with access [to learning] …need to learn more about blended learning models.”

17. Classroom Culture • Praise and the need for improvement were referenced in speaking to Social-Emotional Learning (SEL) and working with at-risk learners. • P28 “The lone thing I would like to see more of with all graduates is more of a focus on social emotional learning.” • Completers demonstrated competence in establishing a climate conducive to meeting the needs of all learners; however, employers felt the increasing complexity of academic and behavioral needs in P-12 classrooms requires a greater understanding by completers on how to meet those needs. • P15 “If you ask her where they are…what skills are deficient and what they need to work on she could tell you…where I think it is falling apart is from paper to presentation.”

18. Relationships • Collegiality, intrapersonal skills and interactions with parents emerged as major categories of praise for completers and, for some, an area where additional training would increase effectiveness. • P11 “I don’t think in general [completers] are confident developing relationships with families. How to handle meetings that go south.” • Completers were frequently cited for both a willingness to collaborate with colleagues and for their contributions to staff initiatives. They were cited as being reflective and open to suggestions from supervisors. • P33 “She is excellent at communicating with faculty and parents, and has integrated herself beautifully this year as a new teacher.”

19. Teacher Identity • Categories that emerged in this theme highlighted the professional nature of completer dispositions and their ability to fit in and contribute to the broader school community. • Almost without exception principals commented on the “growth mindset” of completers in their approach to student learning and their own learning. • P34 “[He] came in with just the sense of ‘this was my starting point but I’m so excited, as a life-long learner , to grow in my craft, to become more and more professional, to learn from my colleagues and peers, to best serve students’.”

20. Results: Employer Interviews • Completer’s impact on student learning: • In her class this year, 89% of ELL met their growth goal and 88% of all students met their goals. • She has made a difference with students. Academically she is really, really, helping children grow. • Her room is amazing. Her test scores are incredible. I believe that she will have the greatest gains in reading, writing, and math of any of our teachers this year.

21. Results: Employer Interviews • Completer’s impact on student learning (cont’d): • Especially in the ELL class that we should not have given him as a first-year teacher. Even with that, he has incorporated all of the SIOP strategies into his instruction. • Yes, student growth goal data show: monitoring Feb-Apr of “intensive” readers’ progress showed 5 of 7 met goals, and the other two were chronically absent. The whole class was at 80% met goals.

22. Using the Data to Inform Practice • Could the strategies learned in the EPP be applied to the classroom contexts of completers? • Best practices learned vs. traditional approaches used • Pre/post assessment vs. assessment strategies in given curriculum • “Trendy and unfounded approaches” • Did the completers have the ability to enact the learned strategies in a manner that is both effective and promotes better learning for all students?

23. Using the Data to Inform Practice • Are employer views of completer attributes and practices similar or different from completer self-reported data? • Training – Pedagogy and management • Classroom Culture – Differentiating for at-risk learners • Do completers have the dispositional qualities of a professional educator and how can we continue to mentor completers in this aspect of professional service?

24. Using the Data to Inform Practice • University supervisor and cooperating teaching meetings • Were the areas of concern beyond what would be expected for a beginning teacher? • Focus groups of completers • Panels of completers provide opportunities for teacher candidates to ask questions about first year of teaching and share experiences. • Faculty ask clarifying questions and record/analyze sessions.

25. Conclusions • The employer viewpoint is a valuable one and adds strength to the overall assessment of program effectiveness given their unique lens. • Time and resource intensive • Geographic area where our completers begin their careers is large • Challenges of tracking completers as they gain employment • location and gathering contact information

26. Questions?