Will that work for us interpreting research from the memphis striving readers project msrp
Sponsored Links
This presentation is the property of its rightful owner.
1 / 88

Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP) PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP). Presented by Ric Potts, MCS; J. Helen Perkins, U of M; Elizabeth Heeren, MCS; Rorie Harris, MCS; and Jill Feldman, RBS 2008 International Reading Association Research Conference Atlanta, GA.

Download Presentation

Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP)

Presented by

Ric Potts, MCS; J. Helen Perkins, U of M; Elizabeth Heeren, MCS; Rorie Harris, MCS; and Jill Feldman, RBS

2008 International Reading Association Research Conference

Atlanta, GA

Session Overview

  • Introduction to the Striving Reader’s grant

  • Overview of Memphis SR research design

  • Year One Impact Analyses

  • Collection of implementation fidelity data

    • implications for practitioners and researchers

  • Planned (Ongoing) Analyses

  • Q & A /Group Discussion

Introduction: Memphis Striving Readers Project (MSRP)

Ric Potts, PI – MSRP

Memphis City Public Schools

Memphis-The City

  • The City of Memphis has a population of 642,251.

  • 63.1% African American

  • 31.3% Caucasian

  • 4.1% Hispanic

And one Elvis

Approximately 70 percent of adolescents

struggle to read. The young people enrolled in

middle and high school who lack the broad

literacy skills to comprehend and learn advanced

academic subjects will suffer serious social,

emotional, and economic consequences.

Reading at Risk: The State Response to the Crisis in Adolescent Literacy, Oct. 2005

Urban Child InstituteThe State of Children in Memphis and Shelby County2006

“Under-educated children have no future.”

Urban Child InstituteThe State of Children in Memphis and Shelby County2006

  • by U.S. standards roughly 75 percent of students in Tennessee fail to meet national grade appropriate standards, and Memphis is at the bottom in Tennessee. . . . Memphis is one of theleast-educated cities in America.

Motivation behindMemphis Striving Readers Project

  • Memphis is among the poorest and least-educated cities in the US

    • 30.1% of all children live in poverty

    • 24.3% of adults have less than a HS education

    • 36.7% have HS diploma or equivalent

    • 30.5% have Assoc. or some college

    • 8.5% have at least a BA

  • MCS is 21st largest K12 district in US >116,000 students

    • Over 95% of MCS’ 196 schools are Title I schools

    • 71% of MCS students qualify for free/reduced price lunch

    • MCS students are 87% AA; 9% White; 4% “other”

    • In 85% of MCS schools, 33% of students change schools during year

    • In 2003-04, the system-wide graduation rate was 61 percent

    • 71% of students in grades 6-8 scored below the 50th percentile on TCAP (Reading/Language Arts)

Striving Readers – A Federal Response

  • In 2005, the Department of Education called for proposals for the Striving Readers grant.

  • In March, 2006, Memphis was one of eight sites awarded the grant.

Memphis Striving Reader Program Targeted Schools

The Whole School Intervention: Memphis Content Literacy Academy (MCLA)

Overview presented by

J. Helen Perkins, SR Co-PI

University of Memphis

A Change Model

A Capacity-Building Model for

Teacher Development

(Cooter & Cooter, 2003)


& Ability

to Coach Others

Refined and



Practice with Coaching

Deeper Learning with

Limited Capacity

First Exposure

No Knowledge

Emphasis: “Deep Training”

(180 hours over two years) …

Memphis Content Literacy AcademyInfusing Simultaneously Across Core Subject Areas Scientifically-based Reading Research (SBRR) Strategies in…


Reading Comprehension

Reading Fluency

Benefits to Teacher – “Laureates”…

  • Advanced Training (180 hours) on scientifically-based reading instruction (SBRR) for urban children

  • A Master Teacher “Coach” to Assist (30 hours) with Implementing New Strategies (in their own classrooms!)

  • Twelve (12) Graduate Semester Hours of Credit from University of Memphis (FREE) (applicable to an advanced degree)

  • Can Seek “Highly Qualified” Endorsement in Reading

  • Books and Materials (FREE)

  • Successin Helping Children Achieve “AYP”

  • Principal Support


Choral Reading

Paired reading

Guided, repeated,

oral reading (pairs)

MCLA Year 1: Selected Strategies

  • Comprehension

  • Question Generation

  • Three- Level Retelling

    • Oral

    • Graphic Organizor

    • Written

  • Comprehension monitoring

  • Expository Text Patterns

  • Multiple Strategies

  • Vocabulary Development

  • Pre-instruction of vocabulary

  • Repeated, multiple exposures

  • Semantic Maps

CREDE Standards

Whole class v. collaborative small group

Reading Next Elements

Classroom Organizational Tools & Strategies: Year 1

  • Use of leveled materials

  • (e.g., National Geographic)

CREDE Formatting of Professional Development Training


Classroom Action Plans (CAPs)Spring 2008Science, Social Studies, & ELAYour task is to develop a series of class lessons where you teach academic vocabulary in a unit of your choice.You must have at least one vocabulary learning strategy/activity that occurs:1. BEFORE students read the assigned text,2. DURING the reading assignment, and3. AFTER the reading assignment

MCLA Classroom Model

  • Gradual release of responsibility

    (teacher modeling, guided practice, independent practice, independent use)

  • Integration of 12 literacy strategies

    (vocabulary, fluency & comprehension)

  • Development of Classroom Action Plans (CAPs)

    (content area lesson plans for strategy implementation including procedures for

    student assessment)

  • On-site support provided by coaches

  • Use of Curriculum Resource Center (CRC) materials

The Principals’ Fellowship

  • Literacy Leadership Practices

  • Real World Problem Solving

  • Create “Literacy Materials Centers”

  • Early Identification w/ Intense/Focused


  • Research-Informed Decision Making

  • Involve Families

  • Needs-Based Scheduling

  • Matching the Most Successful Teachers

    with “Critical Condition” Kids

READ 180, Our Targeted Intervention

Overview provided by

Elizabeth Heeren, SR Grant Coordinator

Memphis City Schools

Program Components

Support materials for differentiated instruction in small group rotation

Tools for student placement and assessment

Student workbooks for Independent Practice in small and whole group rotations

Key Elements of READ 180

  • Fidelity of Implementation

  • 90 minute classes

  • Certified teachers (LA or Reading)

  • District Instructional Support

  • District Technological Support

  • Scholastic training (site-based and on-line)

R180 Correlations to Reading Next Recommendations for Adolescent Literacy

  • Direct, explicit comprehension instruction

  • Motivation and self-directed learning

  • Strategic tutoring

  • Differentiated texts (levels and topics)

  • Technology component

  • Ongoing formative assessment

  • Extended time for literacy

  • Professional development (long-term and on-going)

Memphis Implementation

  • We have 8 schools in the Striving Readers Grant, with up to 120 randomly selected R180 students at each school.

  • Students receive R180 instruction for 2 years.

  • Each student placed in R180 falls in the lowest quartile of TCAP (Reading score).

  • Each student in R180 is paired with a similar student from the lowest quartile who does not receive the treatment (for impact comparison).

MSRP Research Design

Overview presented by

Jill Feldman, SR Research Director

Research for Better Schools

Overall MSRP Goals

To determine:

  • The effects of MCLA on core subject teachers’ knowledge and use of SBRR

    2.The separate and combined effects of MCLA and Read 180 on students’ reading achievement levels, especially students who are identified as struggling readers

    3.The separate and combined effects of MCLA and Read 180 on students’ achievement in core subjects, especially students who are identified as struggling readers

MCLA Program Logic Model


Long-term Outcomes


Funding, staff, curriculum resource center, facilities, incentives, research materials


45 hours of Principal Fellowship participation

100% of principals incorporate plan for using MCLA strategies in SIP

100% attendance of key MCLA events

80% of principals report actively supporting the program

100% of MCLA schools have allocated space for the CRC


90 of hours of MCLA training/yr for 2 years (180 hours)

Engage in weekly coaching sessions or as needed to meet teachers’ differentiated needs

8 CAP “cycles” completed each year for two years

100% of teachers complete performance measures identifying supplemental resources available/those necessary to support content area instruction


50% of students attend 4 classes taught daily by teachers participating in MCLA

Students learn to use 7 of 8 MCLA CAP strategies


Awareness of and interest in staff implementation of MCLA concepts and strategies

Increased advocacy for school-wide use of MCLA strategies


Increased knowledge about MCLA strategies

Improved preparedness to use research-based literacy strategies to teach core academic content

Increased use of direct, explicit instruction to teach research-based comprehension, fluency, and vocabulary strategies in content area classes

Integrated use of multiple MCLA strategies to support ongoing development of content-related instructional units


Increased familiarity with and use of MCLA strategies when engaging with text

Increased internalization of

literacy strategies

Increased confidence engaging with content related texts

Increased interest in school/learning


Improved school climate

School-wide plans include focus on content literacy

Improved instructional leadership


Increased effectiveness supporting students’ content literacy development

Continued collaboration among community of teachers to develop and implement CAPs


Improved reading achievement and content literacy:

10% increase in students scoring proficient in Reading/LA and other subject areas of TCAP

mean increase of five NCEs on ITBS

Increased performance on gateway and EOC exams


Attend 45-hour sessions/yr (2 yrs)

Participate in motivational, recruitment and celebratory events

Discuss MCLA at faculty meetings

Conduct walkthrough observations

Provide opptys for teacher collab

Allocate space for CRC materials


Attend 30 weekly 3-hour MCLA training sessions/yr (2 years)

Develop and implement 8 CAPs per year in collab content-area groups

Meet with coaches for feedback to improve impl of MCLA strategies

Learn to use of leveled texts to support SR content literacy needs


Learn to use MCLA strategies to read/react to content related text (

Higher Quality Teaching

Higher Student Achievement

Study Design MCLA:

Evaluate teacher and student outcomes

experimental design

randomly assigning schools

(to treatment and control conditions)

Teacher outcomes include


frequency of literacy strategy use

Analytic Approach MCLA:

Two-level HLM

spring ITBS and TCAP scores as a function of teacher and school variables

Study Design and Analytic Approach: MCLA

Analytic Decisions

  • Missing Data

    • students missing pretest score(s) deleted from impact analysis on relevant measure(s)

    • teachers missing pretest score deleted from impact analysis on measure

  • Covariates

    • include all student- and school-level covariates in the model

    • run the model

    • eliminate the school covariate with the lowest significance level

      (highest p-value) not less than 0.2

    • repeat steps 2 and 3 until the remaining covariates had p-values less than 0.2

    • repeat steps 2-4 for the student covariates

MCLA: Random Assignment of Schools

Demographic Characteristics of Year 1 MCLA Student Sample

Baseline Comparisons of Students in MCLA Treatment and Control Schools

Selected Characteristics of the Year 1 Teacher Sample for MCLA Impact Analyses

All Variables Included in MCLA Impact Analytical Models for Year 1

READ 180 Logic Model

Study Design:

Evaluate student outcomes using RCT based on random assignment of students to conditions across schools

Student outcome measures:

reading achievement (ITBS)

core content areas (TCAP)

Analytic Approach:

Cross-sectional ITT analyses of reading and core content area achievement

Two-level models using spring ITBS and TCAP scores as a function of student and school variables

R180 Study Design Analytic Approach

READ 180: Enrolled Students

Variables Included in READ 180 Impact Analytic Models (Year One):Dependent and Independent

Variables Included in READ 180 Impact Analytic Models (Year One): Covariates

Year One Impact

Comparison of Teachers in MCLA Treatment and Control Schools on Year-End Indices for Preparedness and Frequency of Use

MCLA Impacts on Students (Year One)

READ 180 Impacts on Students (Year 1)

Collection of Data about Implementation Fidelity

Implications for Researchers and Practitioners

What are our purposes for collecting implementation data?

  • To provide other districts with information about outcomes they might expect when implementing similar interventions with their struggling readers*

  • To set the context for understanding student outcomes

    *Requires MCS to place the needs of the field above local concerns

Reasons to Collect “Double Data”

R180 evaluation is intended to test effects of a

replicable intervention in the real-world:

  • Without the support of external evaluators

  • In ways that emulate what districts will need to do to:

    • monitor implementation

    • obtain process feedback

Reasons to Collect “Double Data”

Collecting data about MCLA and R180 fidelity

  • helps researchers explain patterns of impact findings

  • can be useful in identifying predictors of outcomes

What Is the Role of the Researcher?

  • RBS collects data about:

    • Impact (MCLA & R180)

    • Implementation fidelity

      • To better understand impact or lack thereof

        (MCLA & R180)

      • To support development of MCLA (only)

    • Counterfactual

      • To compare effects to what would have happened in SR schools in the absence of MSRP

What is the Role of MCS?

  • Implement R180 & MCLA

  • Monitor the implementation process

    • Ensure implementation is “on model”

    • Refine service delivery based on formative data

Defining Implementation Fidelity: MCLA

Innovation Configuration Mapping

MCLA Implementation Framework

  • Developing an Innovation Configuration (IC) Map

    (Hall & Hord, 2006)

    • Operationally defines levels of implementation fidelity among clusters of “key active ingredients”

    • Iterative process involving key stakeholders

      • Development team (University of Memphis)

      • Grantee (Memphis City Public Schools)

      • Researchers (Research for Better Schools)

MCLA: Roles & Responsibilities

MCS Administrators:

  • Participate in Principal’s Fellowship

  • Support recruitment and retention efforts

  • Link MCLA w/School Improvement Plan

  • Observe MCLA teachers

    (once/marking period)

  • Allocate space for CRC materials

  • Protect/respect role of coach


  • Design MCLA curricula

    (for teachers & principals)

  • Facilitate writing team activities

  • Meet weekly with instructors (& coaches)

  • Disseminate research about adolescent SR

MCLA Training

Provided by the Developer:

  • 3-hour weekly principal meetings

    (fall;Year 1)

  • 3-hour weekly teacher training sessions per content area

    (180 hours over 2 years)*

  • PD for coaches in

    Mentorship; Urban education; Adolescent lit

    Provided by MCS (coaches):

  • On-site observation of CAPs

  • Model/co-teach strategies

  • Feedback

  • Supplemental resources

    *has included coaches since spring 2007

MCLA Innovation Configuration Map Framework

Instrument Development

With the IC map guiding development, the following

measures were designed to collect data about MCLA


  • Surveys

    • Teacher knowledge about & preparedness to use MCLA strategies

    • Teacher demographic characteristics

    • Teachers’ MCLA Feedback

  • Interviews

    • Principals, coaches, development team, and MCS administrators

  • Teacher Focus Group Discussions

Operationally defining components:“Job Definition”

Aligning the IC Map and Instrument Development: “Job Definition” – Teacher Survey

“Job Definition” - Principal Interviews

MCLA Innovation Configuration Map Framework

Where the rubber hits the “runway”…

MCLA Classroom Implementation

Operationally defining components: Implementation of Lesson Plans

Implementation of lesson plans:Collecting classroom observation data

Implementation of lesson plans:Collecting classroom observation data

MCLA: Implementation Barriers


  • Limited development/planning time

  • Need for coaches with disciplinary content knowledge

  • Challenges in establishing a critical mass of enrolled teachers at each school

  • CRC materials not received until spring 2007

  • Pressure to focus on TCAP test preparation (spring)

  • Difficulty maintaining principal attendance at weekly meetings

MCLA: Planned Implementation Changes


  • Adoption of CREDE (UC-Berkeley) JPA instructional model

  • Reduction in the number of CAPs required of teachers

  • Shortened class schedule/more intensive work with coaches

  • Inclusion of special education teachers among those eligible to enroll

  • Restructured Principal Fellowship

    (includes other school leaders; meets monthly)

Defining Implementation Fidelity: R180

Rorie Harris

Memphis City Public Schools

Findings Related to Implementation

  • Scheduling

    • Scheduling 90 minute blocks in schools using the Middle School concept is difficult. Teams of core content teachers traditionally have 55 minute classes.

    • Interruptions to the 90 minute block occur.

  • Special Education Students

    • READ 180 will only suffice as a SPED student’s intervention if the teacher is SPED-certified.

Findings Related to Implementation

  • Use of Technology

    • Technology issues can negatively affect instructional time.

  • Parents & Students

    • Some parents do not want their children in Reading Intervention classes. They feel like this is a “label.”

    • Classroom management issues impact instruction.

    • Student mobility affects the scope and sequence of reading instruction.

Findings Related to Implementation

  • School Administration

    • Without administrator “buy-in” to the importance of smaller classes and protection of the 90 minute block, fidelity is not supported.

  • Read 180 Teachers

    • It is challenging to encourage ALL teachers to engage in on-line professional development and/or to attend network meetings.

    • Teacher turn-over brings out the need for repeated initial training and reduces the development of teacher leaders.

Indicators of Read 180 Implementation

  • Scholastic identifies several key program aspects

    • Teacher Training/Professional Development

    • Computer Hardware/Software Use

    • Use of Read 180 Materials

    • Group Rotation

    • Class Size

    • Classroom Environment

    • Student Engagement

Sources of Implementation Data

  • Classroom observations during the school year (Fall & Spring)

  • Read 180 program databases (SAM)

  • Enrollment and course-related data from district databases

  • Surveys administered to students (Fall & Spring) and teachers (Spring)

  • Information collected during professional development programs

MCS Data Linked to Implementation Indicators

MCS Data Linked to Implementation Indicators

Overview of Year One Conclusions

Jill Feldman, RBS

(Brief) Conclusions & Discussion

READ 180: No significant Year One student impact

  • Late startup

  • (Most) students will receive two years of intervention

    Planned Future Analyses:

  • Three-level analyses planned to examine whether teacher characteristics exert a moderating effect on student outcomes

  • Exploratory analyses of relationships between amount of

    READ 180 instruction and effects on student outcomes

(Brief) Conclusions & Discussion


  • Significant (moderate) impact on teachers’ frequency and preparedness to use MCLA strategies

  • No significant impact on students’ achievement in reading or core content areas


  • Subjectivity of measure (“Hawthorne Effect”)

  • Teacher findings support program logic model

  • Explore relationship between impact and participation in PD

Next Steps…

Planned Exploratory Analyses

  • Re-run HLM impact analyses to test effects of teacher variables on outcomes

    • Preparedness and use of MCLA strategies

    • Age

    • Experience as teacher (& years at MCS)

    • PD in year prior to MCLA

Planned/ongoing analyses

  • Individual student’s growth over time

  • Rerun HLM with student-level variables

    • # MCLA teachers

    • Student’s school attendance

  • ITS analyses

    • Using TCAP Spring 2003 & 2004 scores

  • Correlating R180 data with TCAP & ITBS

    • for possible use as covariates in HLM

Now It’s Your Turn

  • Ask the panel

  • Share your experiences

    • Triumphs

    • Tribulations

Thank you for joining us!

For additional information contact:

[email protected]

  • Login