Download
1 / 39
390 likes | 457 Views
The Research Design. Research for Better Schools Philadelphia, PA Jill Feldman, Ph.D., Director of Evaluation. What research questions will we ask about MSRP impact ?. Does MCLA effect core subject teachers’ knowledge and use of research-based literacy strategies?
E N D
The Research Design Research for Better Schools Philadelphia, PA Jill Feldman, Ph.D., Director of Evaluation
What research questions will we ask about MSRP impact? • Does MCLA effect core subject teachers’ knowledge and use of research-based literacy strategies? • What are the separate and combined effects of MCLA and Read 180 on students’ reading achievement levels, especially students identified as struggling readers? 3. What are the separate and combined effects of MCLA and Read 180 on students’ achievement in core subjects, especially students identified as struggling readers?
What outcome measures will we use? • Iowa Test of Basic Skills (ITBS) • Vocabulary, Fluency, Comprehension • TCAP • Reading, Social Studies, Science, Mathematics • Gateway and End of Course Assessments • ELA, Mathematics, and Science
What research questions will we ask about MSRP implementation? 1. To what degree do the implemented MCLA & R180 treatments match the intended program standards and features? 2. What contextual district and school level factors may be influencing the implementation of MCLA & R180? 3. How do the professional development events, materials, or structures present in the control schools compare to what is present in the treatment schools?
Research Design for MCLA • 4 matched pairs of schools (N=8) randomly assigned to treatment (MCLA) or control (no MCLA) condition • Content area teachers in cohort 1 to participate in MCLA for Years 1 and 2 • Control group teachers (cohort 2) to participate in MCLA in Years 3 and 4
MCLA: Exploring Efficacy Attempts to address questions about whether or not MCLA can work • Depends upon rapid turnaround of data collected • Relies upon formative feedback to guide program revisions • Requires close collaboration among project stakeholders • To develop measures • To share information and data • To communicate regularly about changes and challenges • To troubleshoot and cooperatively address challenges
Research Design for Read 180TM • Random assignment of “eligible” students enrolled at 8 SR schools, where eligibility means: • No prior participation in READ 180TM • Two or more grade levels behind in reading • Scores in bottom quartile on state assessment (TCAP) • READ 180TM is the treatment • Counterfactual (business as usual*) is the control
Read 180: Exploring Effectiveness Attempts to address questions about whether or not Read 180 will work… • Provides evidence about what happens when R180 is implemented “off the shelf,” (without formative eval support) • Requires MCS to set aside local need for feedback to address questions of importance to field • Establishes a one-way firewall between MCS and RBS
Please review the safety card in the seat pocket… • Balance local knowledge of students’ needs within the identified “eligible pool” without creating selection bias • Address high rates of student mobility • Accurately describe the counterfactual • Obtain parental consent (and students’ assent) to administer the ITBS • Design procedures to prevent crossover • Deal with (inevitable) startup delays
Are the student groups comparable? • Students eligible for READ180 ™: N = 2,277Total students in 8 SR schools: N = 6,170 Students eligible as % of total: 36.9% • No differences in race, gender, ethnicity, or poverty level between conditions • Higher % of ELLs in control group (87 of 1,337 students, or 6.5%) than in R180™ (35 of 940 students, or 3.7%) • Higher % of Sp Ed 8th graders in R180™ group (28.2%) vs control (20.9%)
What, how, and from whom should data be collected? • Use multiple measures and methods • Interview developers, instructors, coaches, & principals • Surveys of teacher knowledge and attitudes • Focus group discussions with teachers • Evaluator observations of PD sessions • Evaluator observations of classroom implementation • Use data to challenge/confirm findings from single sources • Share findings with key stakeholders to determine whether: • data collected are appropriate to support decision making • evaluation findings reflect actual experiences • revisions to the logic model, IC map, and/or instruments are needed
The Flight Plan The MCLA Program Logic Model
Memphis Content Literacy Academy Evaluation Logic Model Outputs Long-term Outcomes Short–term Outcomes Inputs: Funding, staff, curriculum resource center, facilities, incentives, research materials PPrincipals # hours of Principal Fellowship participation # of MCLA events attended Teachers # of hours of MCLA training attended # hours of coaching (contacts) # of CAPS implemented? Observed? videotaped? # of new lesson plans integrating literacy in content area lessons # and type of materials checked out of CRC Students # classes taught by teachers participating in MCLA # MCLA strategies students learn # (freq?) of MCLA strategy use PPrincipals Awareness of and interest in staff implementation of MCLA concepts and strategies Teachers Increased knowledge of MCLA strategies Improved preparedness to use research-based literacy strategies to teach core academic content Increased use of direct, explicit instruction to teach reseach-based comprehension, fluency, and vocabulary strategies in content area classes Integrated use of MCLA strategies to support development of content literacy Students Increased familiarity with and use of MCLA strategies when engaging with text Increased internalization of literacy strategies Increased interest in school/learning • Principals • Improved school climate • School-wide plans include focus on content literacy • Improved instructional leadership • Teachers • Increased effectiveness supporting students’ content literacy development • Continued collaboration among community of teachers to develop and implement CAPs • Students • Improved reading achievement and content literacy: • 10% increase in students scoring proficient in Reading/LA and other subject areas of TCAP • mean increase of five NCEs on ITBS (comprehension? vocab?) Activities Principals Attend four three-hour principal fellowship sessions each year for two (or four?) years Participate in motivational, recruitment and celebratory events Discuss MCLA at faculty meetings Conduct walkthrough observations Provide opptys for teacher collab Allocate space for CRC materials Teachers Attend # weekly MCLA training Develop and implement 8 CAPs per year? Meet with coaches for feedback to improve implementation of MCLA strategies Integrate use of leveled texts to support development of content literacy among struggling readers Students Use MCLA strategies to read/react to content related text (independently? In collaborative groups? Neither? Both?) Higher Quality Teaching & student achievement
Defining what will be evaluated Developing the MCLA Innovation Configuration (IC) Map • Involve diverse groups of stakeholders • The development team • The implementation team (MCS administrators & coaches) • Experienced users • Evaluators • Identify major components of MCLA • Provide observable descriptions of each component • Describe a range of implementation levels
Wheels Up: Resisting Premature Use of “Auto Pilot” With the IC map guiding development, the following measures were designed to collect data a/b MCLA implementation: • Surveys • Teacher knowledge about & preparedness to use MCLA strategies • Teacher demographic characteristics • Teachers’ MCLA Feedback • Interviews • Principals, coaches, development team, and MCS administrators • Teachers Focus Group Discussions
Aligning the IC Map and Instrument Development: “Job Definition” – Teacher Survey
Where the rubber hits the runway… Classroom Implementation
Operationally defining components: Implementation of Lesson Plans
Implementation of lesson plans:Collecting classroom observation data
Implementation of lesson plans:Collecting classroom observation data
Please remain seated with your seatbelts fastened… • Timely turnaround of data summaries • Team meetings to debrief/interpret findings • Testing what you think you “know:” • Productive (& challenging) conversations • Data-driven decision making • Taking Action • Following up (ongoing formative evaluation feedback)
Activity Activity Frequency Frequency Percentage Percentage Coach’s administrative tasks Coach’s administrative tasks 1358 1358 32.2 32.2 Conferencing with teachers Conferencing with teachers 716 716 17.0 17.0 Observation Observation 698 698 16.5 16.5 School administrative tasks School administrative tasks 339 339 8.0 8.0 Collaborative teacher support Collaborative teacher support 330 330 7.8 7.8 Coach’s professional development Coach’s professional development 303 303 7.2 7.2 Assisting teachers in class Assisting teachers in class 138 138 3.3 3.3 Striving Readers evaluation tasks Striving Readers evaluation tasks 138 138 3.3 3.3 Helping teachers prepare Helping teachers prepare 71 71 1.7 1.7 Modeling Modeling 59 59 1.4 1.4 Videotaping/other Videotaping/other 73 73 1.7 1.7 Percentage Distribution of Planned Coaching Activities Logged in Year 1 (N=4,233 entries logged)
Ground Transportation: The Coaching Role Trust b/w coach and teacher(s) is critical: • To provision of CAP implementation support • Pre-conference meeting • CAP Observation • Co-teaching; modeling • Videotapes for use to train teachers, coaches, evaluators • Post observation conference • To effective and strategic selection of CRC & supplemental resources
Avoiding Wind Shear… Team’s unwavering commitment to helping teachers support the success of struggling adolescent readers sum > individual parts
