Download
1 / 64
640 likes | 921 Views
Studying Fidelity of Implementation (FOI): How FOI influenced SCALE-uP’s Theory of Action for Middle School Science Curriculum Materials. *SCALE-uP = Scaling up Curriculum for Achievement Learning and Equity Project, a partnership between George Washington University
E N D
Studying Fidelity of Implementation (FOI): How FOI influenced SCALE-uP’s Theory of Action for Middle School Science Curriculum Materials *SCALE-uP = Scaling up Curriculum for Achievement Learning and Equity Project, a partnership between George Washington University and Montgomery County Public Schools** Sharon Lynch, PI Co-PIs: Curtis Pyke, Joel Kuipers, Michael Szesze** & Bonnie Hansen-Grafton** http://www.gwu.edu/~scale-up/ Prepared for Researchers Without Borders Webinar, May 26, 2010
FOI Research Group • Carol O’Donnell • Suzanne Merchlinski & MCPS evaluation staff • Bonnie Hansen-Grafton • Joelle Lastica • Vasuki Rethinam • Bill Watson • Rob Ochsendorf • Liz Hatchuel • Annie Hansen With special thanks to MCPS middle school science teachers who participated in this study
Disclaimer • Indebted to the Interagency Educational Research Initiative (IERI) administered by the NSF, for research funding for SCALE-uP (7 year research program). • I am currently working at NSF as a Program Director in EHR/DRL ( return to my position as a professor at GWU in September). • However, the ideas and opinions discussed here are entirely my own and in no way represent the those of NSF.
SCALE-uP/FOI Webinar:Cautionary Tale w/ Happy Ending • Background of SCALE-uP and initial Theory of Action • Year 0 Pilot Study: Curriculum modifications are tricky business! • Year 1+: Comparison groups are (incredibly) handy in developing FOI instruments and understanding the study context • Year 2 & 3: Comparison group and FOI evidence are crucial for credible evidence of effectiveness • Year 4: Putting it all together: How the Theory of Action was changed by FOI evidence: FOI as “process” and “structure” constructs for both teacher and student
Background for SCALE-uP and Initial Theory of Action • In 1990’s, AAAS Project 2061 developed a Curriculum Analysis to identify curriculum materials likely to help students learn a target idea (benchmark/standard). • Curriculum Analysis relied on experts’ judgment of written curriculum materials. • Two parts: --Focused, accurate, coherent content on a standard/benchmark --Instructional strategies contained in written curriculum materials
Project 2061 Instructional Strategies for curriculum materials • Convey sense of purpose • Address student ideas and misconceptions • Promote engagement with relevant phenomena • Developing, using scientific ideas • Encourage student thinking • Encourage assessment of progress • Creating positive learning environment: curiosity, all students AAAS. Project 2061.
Background for SCALE-uP • Project 2061 Curriculum Analysis had located only 2 acceptable curriculum units in middle school science. • Units had been field-tested with small numbers of students (no comparison groups). • Note. More mathematics curriculum materials had acceptable ratings and were field-tested and studied and scaled.
Background for SCALE-uP and Research Questions If science curriculum materials having Project 2061 attributes were studied in a series of large (N = ~ 2000) quasi-experiments using carefully matched comparison groups: • Would they be effective? • Would they be equitable? • Would there be a relationship between fidelity of implementation to a unit and student outcomes? • Could the materials be scaled-up in this large school district? • How did they function in classroom (video-ethnography)?
SCALE-uP’s Interventions: 3 Science Units with Coherent Focused on Target Ideas • State of Michigan’s Chemistry That Applies (CTA) focuses on conservation of matter. 8th graders, unit ~ 6 weeks long. • GEMS Lawrence Hall of Science Real Reasons for the Season (Seasons) focuses on the reasons for the Earth’s seasons. 7th graders, unit, ~ 3 weeks. • ARIES Harvard Smithsonian Motion and Forces (M&F) focuses on portions of Newton’s Laws. 6th graders, unit ~ 6 weeks long.
Curriculum Analysis: Instructional Strategies ● =Excellent, ◕=Very Good, ◒=Satisfactory, ◔=Fair ○=Poor
SCALE-uP’s Outcome Measures • Curriculum-independent measure for each unit focusing on the unit’s target idea. • Assessments had good psychometric properties and were developed using a Project 2061 assessment system. • Multiple choice and constructed response items designed to be maximally accessible to students of varied language skills.
Background for SCALE-uP and Initial Theory of Action, c. 2001 Curriculum units highly rated on Curriculum Analysis could be effective overall because: each focused coherently on one big idea/standard/ benchmark each had a carefully planned sequence of activities, and each contained identified instructional strategies leading students to construct understanding of one target idea/benchmark/standard. Big Question: Would they be equitable? Assumption: “Business as usual” comparison classrooms would be less focused, rely more on textbooks and worksheets, and provide less time for guided inquiry and lab work.
“Typical” Theory of Action Teacher Curriculum Materials Student Outcomes
Fidelity of implementationMowbray, Holter, Teague & Bybee, 2003 • “Fidelity of implementation is the extent to which the delivery of an intervention adheres to the original program theory behind its development; it confirms that the implementation of the independent variable in outcome research occurred as planned… • …(and involves) the dynamic nature of fidelity criteria, appropriate validation and statistical analysis methods, the inclusion of structure and process criteria in fidelity assessment and the role of program theory in deciding on the balance between adaptation versus exact replication of model programs.”
SCALE-uP Theory of Action c. 2001 Teacher FOI Instructional Strategies Teacher Curriculum Materials Student Outcomes
SCALE-uP Theory of Actionc. 2001 • Teachers would need to locate and enact the instructional strategies embedded in the curriculum unit (identified via the Project 2061 Curriculum Analysis). • Doing this well would be to “implement with fidelity”. • Hypothesis: The better the implementation of the unit’s embedded instructional strategies in a classroom, the higher the student outcomes. • All SCALE-uP needed to do to create a classroom observation instrument that captured teachers’ implementation of the strategies. • This would become Instructional Strategies Classroom Observation Protocol (more on this later). • Simple!
Questions? On to Fidelity of Implementation (FOI)!
Year 0 (Pilot Study) Results: Chemistry That Applies (CTA) with Modifications Encouraged • CTA’s results showed effect sizes = .52. • Results disaggregated for subgroups of students showed that no students disadvantaged by CTA. • Teachers were asked to modify CTA for diverse learners and record modifications. • Virtually none did! This raised questions about whether to modify CTA in future. Teacher beliefs about the unit were mixed. • Co-PI Szesze wanted to be sure units were unambiguously effective, or not. • Decision to “implement with fidelity” in the future studies . • Teachers and researchers drew up fidelity guidelines together.
Lesson Learned :Modifying Curriculum Unit is Tricky Business! • If an intervention does not have solid evidence of efficacy/effectiveness, then modifications muddy the water; CTA and the other units were “unproven” interventions. • Therefore, their critical components could not be “known”, but assumed. • In retrospect, this was a good decision for SCALE-uP; capturing modifications while trying to establish if a unit “worked” would not be credible in this school district context.
Iterative Process of Identifying Critical Attributes, Measuring Attributes, and Looking for Relationships between FOI and Outcomes Identify Critical Attributes of Intervention FOI measure Reconsider Critical Attributes Outcomes Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3),315-340.
Questions? On to Year 1 Replication of CTA Building the first FOI Instrument
Yr. 1 Replication of CTA and Develop ISCOP/FOI Process • Given Theory of Action (the more teachers implemented Project 2061 instructional strategies embedded in curriculum units, the higher student outcomes), we began to develop an instrument that could capture fidelity to identified Project 2061 instructional strategies. • A “generic” instrument for all 3 units studied, but the units had been carefully vetted and had much in common. • ISCOP (Instructional Strategies Classroom Observation Instrument), the first FOI measure was born, and developed over the next 4 years.
Years 1, 2, 3, 4: Test ISCOP in Treatment and Comparison Classrooms • Was the ISCOP capturing fidelity to instructional strategies in a way that discriminated between Treatment and Comparison classrooms? • ISCOP did not discriminate very well, so it was refined and refined and refined. • Was ISCOP a “bad” instrument? Or were Treatment and Comparison classrooms similar for Instructional Strategies? • If Treatment and Comparison classrooms were similar in strategies, should the Theory of Action be revisited?
Lesson Learned: Comparison Classrooms Invaluable as Counterfactuals • ISCOP data suggested that more complexity than assumed in the Theory of Action. • Measuring FOI relying solely with ISCOP might not answer FOI research question if comparison classroom data were taken into consideration. • Generic measures of instructional strategies/FOI process are notoriously hard to develop—valid and reliable?
Questions? On to Years 2 and 3 with a new unit, Real Reasons for the Seasons (Seasons)
Year 2 and 3 Results: Comparison group outscores Seasons group: How to account for this? • Seasons studied in 7th grade classrooms (N ~ 2000 students and ~ 40 classrooms). • Year 2 and 3 results: Comparison group has higher outcome than Seasons on curriculum independent measure (ES = -.36, -.18, respectively).
Comparison Classrooms Invaluable as Counterfactuals • Comparison classroom teachers surveyed and interviewed. • Data showed that Comparison classrooms were: -Non-traditional -Variety of curriculum materials used, including other inquiry units -Focused on the target idea -Equal Duration of Treatment and Comparison units -Comparison and Treatment teacher characteristics similar.
Seasons Replication in Year 3: Focus on FOI • Use ISCOP in Treatment and Comparison classrooms. • Develop a new Lesson Flow FOI process measure that gauged Teacher, Student-Group or Individual Centeredness of classrooms based on hunch that Student-Group Centeredness was important to student construction of science ideas in groups. • Videotaped a Seasons and Comparison classroom. • Interviewed and surveyed Seasons and Comparison Teachers again.
Year 3 FOI Results • Only 3 items on ISCOP showed significant differences between Seasons and Comparison classrooms, and 2 of 3 were observed more often in Comparison classrooms. • Lesson Flow: Seasons classrooms were more teacher-centered than Comparison: Teacher-Centeredness Seasons = 71 % of time Comparison = 58 % of time • Video-data backed this up. • Emerging Conjecture: Students need time in to work and talk in groups to develop their ideas and Seasons allowed less time for this.
SCALE-uP Theory of Action c. 2006 Teacher FOI Instructional Strategies Teacher Curriculum Materials Student Outcomes Student FOI Student Group Centeredness
Changing Theory of Action and Approach to FOI c. 2006 • Focusing FOI entirely on teacher in a guided inquiry unit may be simplistic. • Student agency implicated in FOI constructs. • But Lesson Flow is controlled by the teacher who provides students with the time/space to work in groups; students can choose to do the science work, or not. • Lesson Flow is not Time On Task.
Seasons Unit and FOI:The Research Re-visited • Seasons was designed to be a supplementary unit. • Developers seemed not to understand the implications of this when agreement was made to study effectiveness/FOI. • SCALE-uP researchers did not appreciate the differences in philosophy of Seasons with research design. • Seasons was a bad match for this study. • Revisions made to Seasons based upon feedback from MCPS teachers.
Questions? On to Year 4 and Motion and Forces (M&F) Note this is the third curriculum unit studied, not to be confused with the first two, and was implemented in 6th grade classrooms.
M&F Results in Years 2 and 3:Underwhelming • Year 2 ES = +.10 • Year 3 ES = -.06 • FOI had not been emphasized, so there was little information to explain lackluster outcomes. • However, we learned that students had not been issued M&F student Journals in Years 2 and 3. Was this an important, overlooked FOI issue?
Year 4 Replication of M&F: Focus on FOI • Replicated quasi-experiment in 10 new schools (N ~ 2000 students). • Teachers asked to focus on FOI. • Wonderful teachers who understood the study goal: study the impact of M&F.
FOI in Year 4 with M&F FOI included: • ISCOP (Process FOI) • Lesson Flow (Process FOI) • Adherence to unit (ACOP), a new instrument that measured close adherence to M&F’s structure (Structure FOI) • Teacher Interviews/Surveys • Teacher Logs • Student Journal Entries, # of responses to Journal (Structure FOI) • Student survey about self-reporting use of instructional strategies (Process FOI)
Overall Results for M&F in Year 4 • Student Level results using traditional ANOVA, ES = .23 • Classroom Level results using HLM, ES = .56 (Rethinam, Lynch, & Pyke, 2008)
ISCOP Strategies: Means and Correlations with Outcomes for M&F
ISCOP • This “generic” instrument for measuring FOI instructional strategies did not obviously distinguish between M&F and Comparison classrooms. • But some items were correlated with student outcomes for M&F classrooms. • O’Donnell dissertation elegantly teased out which instructional strategies seemed to matter for higher student outcomes. • ISCOP needs more work on validity and reliability--OR SCALE-uP Theory of Change needs refinement—Project 2061 Curriculum Analysis needs refinement?
Lesson Flow for Instruction for units on Motion and Force M&F Comparison
Lesson Flow • M&F classrooms provided more time for students to work in groups and individually than Comparison classrooms. • Students construct meaning in groups, consistent with notion of community of practice and situated cognition.
Adhering to M&F Lesson Components (ACOP) • M&F teachers adhered to M&F unit > 80% of time. • ACOP did not predict outcomes because the range was narrow; this was a good thing for this study because high fidelity in this measure of “structure” is credible and strong—good face validity for FOI, and highly reliable—the unit was well-implemented. • Teachers adhered to the unit for this study, to ascertain impact of M&F. • Thank you teachers.
