Nancy Romance, Florida Atlantic University Michael Vitale , East Carolina University

1. Considering Fidelity as an Element Within Scale Up InitiativesValidation of a Multi-Phase Scale Up Design for aKnowledge-Based Intervention in Science andReading Comprehension Nancy Romance, Florida Atlantic University Michael Vitale, East Carolina University Jerry Haky, Florida Atlantic University NSF IERI PI Meeting Washington, DC August 25-26, 2005

2. Project Intervention Description/Overview • Science IDEAS: Grades 3-5 (12 Elementary Schools) • Replaces Reading/Language Arts with 2 hrs/day of in-depth science • Variety of activities (hands-on, reading, concept mapping, writing) linked to conceptual content taught • Emphasis on students learning more about what they are learning • Science IDEAS: Grades 6-8 (6 Middle Schools) • Teacher strategies that emphasize core science concepts • Student strategies that emphasize reading with comprehension and concept mapping to organize knowledge • Emphasis on students learning more about what they are learning

3. Approaching Scale Up as an Evolution... • Transferring responsibility for implementation from researchers to school system • Building capacity for school system implementation • Specialized expertise • Organizational infrastructure • Maintaining fidelity of implementation and achievement outcomes as success criteria re: • Original research results • Initial school system implementation • Sustainability and expansion • Establishing the “value added” by intervention to school system

4. Instructional Systems Development (ISD) as Framework for Scale Up • ISD methodology specifies requirements for development re: • Designing effective instructional systems • Intervention/delivery component • Management component • Engineering of implementation requirements • Criteria/processes for establishing effectiveness • ISD requirements logically parallel those for scale up • Reverse engineering of ISD methodology yields a framework for researchers pursuing scale up • Guide for addressing key scale up elements • Framework for scale up research • ISD framework provides contextual linkage for fidelity re: implementation vs. scale up

5. Project Approach to Fidelity of Implementation • Maintain focus on fidelity as school capacity is developed and transfer of responsibility occurs • Instrumentation for monitoring fidelity • Focuses on Science IDEAS (3-5) implementation requirements • Used for judgments re: whether requirements are being met (not at all, partial, with fidelity, outstanding fidelity) • Data obtained 3 times per year • Three fidelity perspectives • Teachers: Detailed self-report • Principals: Clinical judgment (short/global form  informal classroom/grade level planning observations as context) • Project Staff: Clinical judgment (short/global form  informal classroom / grade level planning/training observations as context)

6. Project Use of Fidelity Information • Teacher-level data entered into database • Data reported to principals: • Grade level summary (w/o teacher names) • Trends across year by grade to track fidelity • Average of ratings used as predictor(s) of student outcomes* • Collaborative discussions by principals/project staff re: implementation strengths/weaknesses • Teacher self-report + clinical judgments used to adjust professional development / support components * Priority for 2005-2006 re: 2003-2005 data

7. Fidelity of Implementation Trends: 2004-2005 Percent of teachers by fidelity of implementation categories for Science IDEAS in grades 3-4-5 ---------------------------------------------------------------------------------------------------------- Fidelity Rating 1st 9 Wks. 2nd 9Wks. 3rd 9Wks. ---------------------------------------------------------------------------------------------------------- 0 (No Implementation) 10 9 4 1 (Partial Implementation) 35 27 24 2 (Implementation with 47 49 49 Fidelity 3 (Outstanding Implementation) 8 15 24 ---------------------------------------------------------------------------------------------------------- Note 1- N observed teachers = 183, 131, 154 for 1st, 2nd , and 3rd 9-weeks, respectively. Note 2: Implementation with fidelity ratings increased 55% to 64% to 73%. A preliminary OLS 3-factor ANOVA with two between factors (Grade Level, New vs. Continuing School) and one within factor (Time of Rating) of the percent of teachers in fidelity category 2 or 3 revealed a significant linear trend across the school year, with no other main effects or interactions effects found significant. This finding showed that the degree of fidelity obtained with the six new elementary schools matched that of the four continuing schools. Also, the comparable percentage for 2003-2004 was 65%.

8. Fidelity of Implementation Trends: 2004-2005 Mean minutes of daily instructional time allocated to Science IDEAS in new and continuing schools in grades 3-4-5 ---------------------------------------------------------------------------------------------------------- School Category N Schools Grade 3 Grade 4 Grade 5 Mean ---------------------------------------------------------------------------------------------------------- Continuing 4 77 98 90 88 New 10 72 79 99 83 ---------------------------------------------------------------------------------------------------------- Mean 74 88 94 85 Note 1- N observed teachers = 183, 131, 154 for 1st, 2nd , and 3rd 9-weeks, respectively. Note 2- Preliminary OLS 2-factor ANOVA (Grade Level, New vs. Continuing) found the main effect for Grade statistically significant (with significance due to a linear trend), but paralleling the preceding analysis for fidelity (Table 1), no difference was found between Continuing and New schools (nor was the Grade x Category interaction significant). Thus, the level of implementation of new schools matched that of continuing schools (a positive finding for the scale up component). But, the average amount of time allocated for Science IDEAS was well below the 2-hour standard set by the project. Given this finding, the enhancement of the scale up component to address this issue is a major project priority for 2005-2006.

9. Using Fidelity as an Engineering Toolfor Scale Up: Major Priorities for 2005-2006 • Providing principals with more rapid feedback in the form of trends • Principals need fidelity trend data for instructional management • Fidelity data/feedback system necessary for scale up • Including area/central administrators in fidelity “information loop” • Area/central administrators need fidelity trend data by school as a management tool • Including administrators in “loop” necessary for scale up • Developing statistical models linking fidelity to achievement projections for principals/administrators • Such models provide dynamic support for management • Achievement projections important for scale up

10. Addressing Session Fidelity Questions within the Context of Scale Up • Fidelity working definition / how measured • Fidelity is a direct measure of implementation • Try to capture the process through multiple perspectives • Major fidelity research questions • Can the rate of fidelity be accelerated for schools new to the project (vs. older schools)? • Can fidelity be sustained for schools implementing the model? • To what degree is fidelity predictive of student outcomes?

11. Addressing Session Fidelity Questions within the Context of Scale Up • What did you learn about measuring fidelity and (4) what were challenges? • For scale up, fidelity assessment must focus on teachers, principals, and area/central administrators • Each focus has a different set of fidelity criteria • Teachers- Quality implementation of the intervention (e.g., Science IDEAS) • Principals- Leadership and support of classroom implementation • Area Administrators- Leadership and support of school implementation • District Curriculum Administrators- Leadership and support for the intervention

12. Addressing Session Fidelity Questions within the Context of Scale Up Major challenges: • Project overhead associated with the degree of interaction necessary to develop the specialized expertise of school personnel to make sound fidelity judgments in scale up • Principals about teacher fidelity • Area administrators about school fidelity • Curriculum administrators about intervention fidelity • Project overhead associated with project staff monitoring teacher/principal fidelity for implementation research • Project overhead to provide the systemic motivation for principals to monitor fidelity and use fidelity information for instructional decision-making • Project overhead to utilize fidelity perspectives to help establish the “ value” of the intervention for raising achievement expectations

13. Addressing Session Fidelity Questions within the Context of Scale Up • What is the relationship among different views of fidelity? • Science IDEAS is an instructional system with six key elements that allows teachers a great deal of flexibility (i.e., given an instructional focus, select activity elements to involve students in learning more...) • With regard to classroom implementation fidelity, teachers, principals, and project staff have the same perspective

14. Addressing Session Fidelity Questions within the Context of Scale Up • What are the implications for scale up of teachers being monitored re: fidelity? • Our goal is to maintain fidelity of implementation as the implementation/monitoring responsibilities are transferred from project staff to principals (and administrators) • So monitoring fidelity is a continuing element in the management of the implementation rather than an artificial process that would be discontinued • Does fidelity relate differently to implementation vs. scale up studies? Fidelity of implementation issues are the same for implementation and scale up studies- what is different is who has the responsibility

15. Addressing Session Fidelity Questions within the Context of Scale Up • Are you able to link fidelity with student outcomes? Informally we have seen a relationship (just looking at data by teacher). We have the data to do so and are in the process of analyzing the data to be able to explore linkage between fidelity and outcomes

16. Considering Fidelity as an Element Within Scale Up InitiativesValidation of a Multi-Phase Scale Up Design for aKnowledge-Based Intervention in Science andReading Comprehension Nancy Romance, Florida Atlantic University Michael Vitale, East Carolina University Jerry Haky, Florida Atlantic University NSF IERI PI Meeting Washington, DC August 25-26, 2005