PROMISE Research Findings and Lessons Learned

1. May 20, 2010 Welcome to the PROMISE Videoconference Report on Research findings. Welcome to the PROMISE Videoconference Report on Research findings.

2. Purpose of Presentation • To share the findings and conclusions from research studies of the PROMISE pilot initiative PROMISE, standing for: Pursuing Regional Opportunities for Mentoring, Innovation, and Success for English Learners, has been an unprecedented six-county collaboration that initially included Los Angeles, Orange, Riverside, San Bernardino and San Diego County Offices of Education - and was joined by Ventura County Office of Education. It has now been almost 8 years since the initiative was first conceived by the county office superintendents who recognized the urgency represented by the widespread undereducation of English Learners They recognized then that by collaborating on a joint initiative, the counties might mount an initiative that would significant accelerate and sustain English Learner achievement -- creating a model not only in our own region, but across the nation. Over years of planning and three years piloting the PROMISE Model, we are pleased today to brief YOU, the Working Group, on the findings of the four research studies that together documented the work of the pilot and the impacts upon students, teachers, leadership and schools. The findings will be released to the PROMISE Initiative sites and districts on February 10th. This videoconference is designed to serve three purposes: (see slide)PROMISE, standing for: Pursuing Regional Opportunities for Mentoring, Innovation, and Success for English Learners, has been an unprecedented six-county collaboration that initially included Los Angeles, Orange, Riverside, San Bernardino and San Diego County Offices of Education - and was joined by Ventura County Office of Education. It has now been almost 8 years since the initiative was first conceived by the county office superintendents who recognized the urgency represented by the widespread undereducation of English Learners They recognized then that by collaborating on a joint initiative, the counties might mount an initiative that would significant accelerate and sustain English Learner achievement -- creating a model not only in our own region, but across the nation. Over years of planning and three years piloting the PROMISE Model, we are pleased today to brief YOU, the Working Group, on the findings of the four research studies that together documented the work of the pilot and the impacts upon students, teachers, leadership and schools. The findings will be released to the PROMISE Initiative sites and districts on February 10th. This videoconference is designed to serve three purposes: (see slide)

3. Laurie Olsen, PROMISE Design Consultant and researcher Magaly Lavadenz, Loyola Marymount University Presenters

4. PROMISE: a collaborative across a six county region This shows the scope of the project. Statistics on percentage of English Learners in the state. More than 60% of California English Learners are in this six-county region, and 20% of English Learners in the nation are in these six counties. PROMISE originated in 2002 at the request of five of the six county superintendents and Ventura joined in 2005. This shows the scope of the project. Statistics on percentage of English Learners in the state. More than 60% of California English Learners are in this six-county region, and 20% of English Learners in the nation are in these six counties. PROMISE originated in 2002 at the request of five of the six county superintendents and Ventura joined in 2005.

5. • Initiated in 2002 by 5 County Superintendents concerned with EL academic achievement in the region - joined in 2005 by Ventura • A common vision for EL success was developed, and research-based guiding principles were distilled; • A joint commitment to design and pilot a process for enacting core principles in local contexts, including an infrastructure to support implementation. Introduction to Pilot Study History As context for the research findings, I want to share with you a bit of the history. Joined together by a sense of urgency about English Learner underachievement, the leadership of the six county offices of education worked first to distill a common vision for EL success, and then to distill from the research on effective English Learner practices, a set of guiding principles. These were the framework for the development of a PROMISE Model. The county offices commited to working together to design and pilot that model, and to provide a powerful infrastructure that might support schools in implementing the model.As context for the research findings, I want to share with you a bit of the history. Joined together by a sense of urgency about English Learner underachievement, the leadership of the six county offices of education worked first to distill a common vision for EL success, and then to distill from the research on effective English Learner practices, a set of guiding principles. These were the framework for the development of a PROMISE Model. The county offices commited to working together to design and pilot that model, and to provide a powerful infrastructure that might support schools in implementing the model.

6. Three-year PROMISE PILOT (ended June 2009) 6 Districts, 14 Schools- Pre-K - 12th Grade The six county offices, plus partner organizations California Tomorrow and Loyola Maryount University, formed the core of the infrastructure of support and leadership for the PROMISE Initiative. Each county office selected ONE school district and worked with 2 or 3 sites from each district intensively, so that, in aggregate, the Initiative encompassed a “virtual district” of 15 schools ranging from Pre-K (two sites), elementary schools, middle schools and high schools. The six county offices, plus partner organizations California Tomorrow and Loyola Maryount University, formed the core of the infrastructure of support and leadership for the PROMISE Initiative. Each county office selected ONE school district and worked with 2 or 3 sites from each district intensively, so that, in aggregate, the Initiative encompassed a “virtual district” of 15 schools ranging from Pre-K (two sites), elementary schools, middle schools and high schools.

7. The PROMISE Sites: Demographic Context Together, the 6 counties serve over 2/3 of the ELs in California. (see slide) Because of the commitment to contribute to the field of EL education, documentation and research were deemed highly important.Together, the 6 counties serve over 2/3 of the ELs in California. (see slide) Because of the commitment to contribute to the field of EL education, documentation and research were deemed highly important.

8. Laurie Olsen - the model, the pilot, the work of the schools over the 3 years, and impacts on schools Loyola Marymount University (Magaly Lavadenz, Elvira Armas) - impacts on classroom teaching practices Loyola Marymount University (Franca Dell’Olio) - impacts on principals Kathryn Lindholm-Leary - impacts on students A set of 4 research studies

9. The PROMISE Model - the PROMISE Pilot

10. The PROMISE Theory of Change and Theoretical Model

11. Safe and affirming environment Empowering pedagogy Challenging and Relevant Curriculum High quality instructional materials Valid assessment systems High quality professional development Family and Community Engagement Advocacy Oriented Leadership The 8 Core Principles

12. • County Offices of Education • PROMISE Design Center • PROMISE Facilitators • PROMISE Lead Teams • PROMISE Partners • PROMISE sites and district leadership • The PROMISE Networks!

13. • Reasons for participation: inspiration of vision, sense of urgency, credibility of the County offices, and the promise of support, to be part of a community of schools; • Not always “voluntary” or an informed choice Assurances signed - but not always followed; • The application and selection process itself was important. Recruitment, application, selection of sites

14. Year One: Selecting core principles, developing a plan, and answering: “What IS PROMISE?

15. Annual “mid year” symposia Beginning and end of the year district retreats for Lead Teams Support for networking across schools Lead Team meetings to guide strategy, monitor implementation of the plan, evolve the work Coaching, meetings, problem-solving support for leadership through Design Center and Working Group Access to researchers Formation of working groups across sites

16. • Implementation of plans, reflection, deepening the work • Professional development tended to be similar across sites • Lead Teams faced challenges and obstacles • Degree of implementation varied • From work on a few principles to more • Advocacy oriented leadership emerged Years 2 and 3

17.

18. • Unfamiliar approach - required learning • Meaning-making is a process • Some principles were more challenging than others (e.g., empowering pedagogy, relevant curriculum, advocacy oriented leadership) • Led to research-based practices - and new exemplars of all principles! • Created sense of coherence, reduces sense of fragmentation and overwhelm • Each principle opened a door to the others • Do not stand alone! Lessons Learned: Core principles

19. • Plans deepened and became more research-based over time • External lens of partners, Working group and Design Center were helpful • Development of protocols, tools, processes for reflection were important • Relied on trust and forums where people could work together across roles • Highly valued by participating schools Lessons Learned: Co-design and reflective practice

20. • Degree of engagement with and implementation of the PROMISE model made a difference in number and depth of research-based changes made in schools; • The more deeply schools engaged in the model, the more comprehensively their work addressed needs of ELs; • PROMISE engagement built and strengthened leadership at the sites - building more collaborative responsibility for EL program and success. IMPACT: Changes in Schools

21. Magaly Lavadenz, Ph.D & Elvira Armas, Ed.D. Loyola Marymount University Center for Equity for English Learners (CEEL) Did PROMISE impact classroom teaching practices? • Descriptive/observational research method • Mixed-methods design (Triangulated Data Collection) - Quantitative data - observations using the OPAL instrument - Qualitative data - semi-structured interview protocols • Research questions - What are teachers’ current practices in instruction of ELs? - How do these practices reflect current research on effective instruction of ELs as measured by the OPAL? - What are teachers’ perceptions of current practices for meeting the needs of ELs? What professional development do they still need? • 381 classroom observations 78 ES – 145 MS – 158 HS • 177 teacher interviews 34 ES – 72 MS – 71 HS • Descriptive/observational research method • Mixed-methods design (Triangulated Data Collection) - Quantitative data - observations using the OPAL instrument - Qualitative data - semi-structured interview protocols • Research questions - What are teachers’ current practices in instruction of ELs? - How do these practices reflect current research on effective instruction of ELs as measured by the OPAL? - What are teachers’ perceptions of current practices for meeting the needs of ELs? What professional development do they still need? • 381 classroom observations 78 ES – 145 MS – 158 HS • 177 teacher interviews 34 ES – 72 MS – 71 HS

22. Observation InstrumentThe OPAL (Observation Protocol for Academic Literacies)

23. OPAL: Year 1, Year 2, and Year3Overall OPAL (Four Domains) Overall when answering the question of how much effect the program yielded… statistical results reveal small to moderate effects 1) small effect at the .10 level; 2) moderate effect at the .30 level; and 3) large effect at the .50 level. There are several possible reasons for these results. It may be that the implementation efforts were not widespread within each of the school sites and across the PROMISE Initiative sites. This was ascertained through the analyses of qualitative results collected from the sample population, which included a purposeful sampling from multiple grade levels and various instructional program types at each of the schools. Furthermore, teacher interview findings revealed that there was inconsistent clarity about PROMISE, professional development efforts linked to PROMISE, and how PROMISE impacted classroom instruction. For the K-5 subgroup, results show that there is some sustainability in practices. Mean scores increased, or maintained, from Year 1 to Year 3 for all domains except Connections. For the 6-8 subgroup, scores shows that there is some sustainability in practices observed in grades 6-8 in only two domains. Mean scores increased, or maintained, from Year 1 to Year 3 for Comprehensibility and Interactions. The Interactions domain shows a slight increase in scores across years for this subgroup. For the 9-12 subgroup, results show that there is no significant change in observed practices over time. Mean scores from Year 1 to Year 3 increased slightly for Interactions and ConnectionsOverall when answering the question of how much effect the program yielded… statistical results reveal small to moderate effects 1) small effect at the .10 level; 2) moderate effect at the .30 level; and 3) large effect at the .50 level. There are several possible reasons for these results. It may be that the implementation efforts were not widespread within each of the school sites and across the PROMISE Initiative sites. This was ascertained through the analyses of qualitative results collected from the sample population, which included a purposeful sampling from multiple grade levels and various instructional program types at each of the schools. Furthermore, teacher interview findings revealed that there was inconsistent clarity about PROMISE, professional development efforts linked to PROMISE, and how PROMISE impacted classroom instruction. For the K-5 subgroup, results show that there is some sustainability in practices. Mean scores increased, or maintained, from Year 1 to Year 3 for all domains except Connections. For the 6-8 subgroup, scores shows that there is some sustainability in practices observed in grades 6-8 in only two domains. Mean scores increased, or maintained, from Year 1 to Year 3 for Comprehensibility and Interactions. The Interactions domain shows a slight increase in scores across years for this subgroup. For the 9-12 subgroup, results show that there is no significant change in observed practices over time. Mean scores from Year 1 to Year 3 increased slightly for Interactions and Connections

24.  Rigorous & Relevant Curriculum • Theme 1: Reliance on a Prescriptive Curriculum - Increasing efficacy over course of the pilot - Instructional decisions primarily guided by pacing guides and grade level standards - Limited use of supplemental instructional materials (some increase over 3 years) - Content driven by core materials - Students primarily engaged in lower-level teaching and learning activities Limited evidence of student-generated goals - Teachers developed skills, courage, efficacy to adapt curriculum to student needs Rigorous and Relevant Curriculum: K-12 mean scores and standard deviation reported as Year 1- 2.72 (.85); Year 2- 3.29 (1.11); and Year 3- 2.93(0.95) Teacher interview data revealed that teachers pointed to pacing guides and grade level standards as driving force for making instructional decisions about curriculum Few teachers mentioned the use of supplemental instructional materials as resources for planning and delivering instruction Fewer mentioned the use of primary language resources to support and supplement Content was consistently driven by core materials and primarily engaged students in low-level teaching and learning activities While many classrooms listed standards and expectations, few provided evidence of standards and goals listed in student-language so that they were easily comprehensible to ELs Objectives were typed and posted, but not visible to students There was limited evidence of student-generated goals and long-term planning evident for classroom instruction Over three year implementation period there was a slight increase in the use of supplemental materials Sample Teacher Comment “Decisions about curriculum, I’m not able to do this. I am given a pacing guide.” Sample Anecdotal Notes: “We are practicing our sounds today. We haven’t been able to put them into syllables so we are practicing because we need it for testing.” – Elementary school teacher “Here are the answers to the problems [math] that you had for homework. (Teacher standing at the front of the room writing answers to problems on the white board. Students correct paper and turn it in). Now, get ready for lesson 8.1. Our objective for this lesson is in your book and is on adding unlike fractions.’’ – Elementary school teacher “Today our objective is to analyze the Feudal way of life and the importance of the warrior code. What were the names? What is the code? Find the name of the code. Write what is meant by the Samurai code.” – Middle school teacherRigorous and Relevant Curriculum: K-12 mean scores and standard deviation reported as Year 1- 2.72 (.85); Year 2- 3.29 (1.11); and Year 3- 2.93(0.95) Teacher interview data revealed that teachers pointed to pacing guides and grade level standards as driving force for making instructional decisions about curriculum Few teachers mentioned the use of supplemental instructional materials as resources for planning and delivering instruction Fewer mentioned the use of primary language resources to support and supplement Content was consistently driven by core materials and primarily engaged students in low-level teaching and learning activities While many classrooms listed standards and expectations, few provided evidence of standards and goals listed in student-language so that they were easily comprehensible to ELs Objectives were typed and posted, but not visible to students There was limited evidence of student-generated goals and long-term planning evident for classroom instruction Over three year implementation period there was a slight increase in the use of supplemental materials Sample Teacher Comment “Decisions about curriculum, I’m not able to do this. I am given a pacing guide.” Sample Anecdotal Notes: “We are practicing our sounds today. We haven’t been able to put them into syllables so we are practicing because we need it for testing.” – Elementary school teacher “Here are the answers to the problems [math] that you had for homework. (Teacher standing at the front of the room writing answers to problems on the white board. Students correct paper and turn it in). Now, get ready for lesson 8.1. Our objective for this lesson is in your book and is on adding unlike fractions.’’ – Elementary school teacher “Today our objective is to analyze the Feudal way of life and the importance of the warrior code. What were the names? What is the code? Find the name of the code. Write what is meant by the Samurai code.” – Middle school teacher

25. Connections • Theme 2: Limited Opportunities to Make Connections - Few engaged students in critical thinking about subject matter to make it meaningful Few included opportunities to compare things from the past and consider cultural traditions Few opportunities to make cross-curricular connections K-12 mean scores and standard deviation reported as Year 1- 2.76 (1.06); Year 2- 3.07(1.20); and Year 3- 2.80 (1.10) Sample Teacher interview comments “I can relate to their experiences at home. I look for current events everywhere so they can be motivated to want to learn about history. The discussion starts when you are able to build the relationship with the students.” – High school teacher “It is powerful to be a Latina teacher. We have discussions in the classroom and they are able to connect.” – Middle school teacher “I bring back concepts they learned in other grade levels and talk about how they are just expanding this to the fourth grade.” – Elementary school teacher Sample Anecdotal Notes: “This is important. It will be on the test.” – High school teacher “You need to learn this so that you can do well on the CST (California Standards Test).” – Middle school teacher “When you go to the next level [book] you will need to know this.” – Elementary school teacherK-12 mean scores and standard deviation reported as Year 1- 2.76 (1.06); Year 2- 3.07(1.20); and Year 3- 2.80 (1.10) Sample Teacher interview comments “I can relate to their experiences at home. I look for current events everywhere so they can be motivated to want to learn about history. The discussion starts when you are able to build the relationship with the students.” – High school teacher “It is powerful to be a Latina teacher. We have discussions in the classroom and they are able to connect.” – Middle school teacher “I bring back concepts they learned in other grade levels and talk about how they are just expanding this to the fourth grade.” – Elementary school teacher Sample Anecdotal Notes: “This is important. It will be on the test.” – High school teacher “You need to learn this so that you can do well on the CST (California Standards Test).” – Middle school teacher “When you go to the next level [book] you will need to know this.” – Elementary school teacher

26. Comprehensibility • Theme 3: Increase in Targeted Efforts for Comprehensible Input and Output - Increased implementation of research-based strategies - Intentional use of graphic organizers from focused PD efforts - Increased use of realia and/or visuals - Common strategies for monitoring comprehension: Q & A, walking around, checking papers, use of whiteboards - Increased number of teachers MS/HS classrooms using note-taking and checking strategies - Lack of clarity around use of primary language K-12 mean scores and standard deviation reported as Year 1- 3.33 (1.05); Year 2- 3.85 (1.26); and Year 3- 3.54 (0.98) Increased implementation of research-based strategies to promote comprehensibility Use of graphic organizers stemming from teacher participation in Thinking Maps Training, Project WRITE, and Project GLAD evident in school sites that reported receiving professional development in these programs Teachers were observed using realia and/or visuals such as pictures, pictorial input charts, diagrams, and overhead transparencies Question and answers were also reported as effective strategies, followed by monitoring comprehensibility through informal assessments by walking around the class, checking students’ papers, or using whiteboards to ask students to show their understanding of a concept or skill Over three-year observation period, an increased number of teachers in the middle and high schools were observed using note-taking and checking strategies to monitor comprehension Primary language use Concerns about when and how to use primary language to clarify learning and check for understanding Sample Anecdotal Notes: “Thinking Maps posted. Teacher generated map and students provided input for map. Students writing sentences based on Thinking Map.” – Elementary school classroom “Pictorial Input charts posted in Spanish and English. Students placed index cards over targeted vocabulary.” – Elementary school classroom “Concept Map used for lesson. Teacher poses question: ¿Qué mas puedes decir de este concepto? (What else can you state about this concept?)” – Middle school classroom K-12 mean scores and standard deviation reported as Year 1- 3.33 (1.05); Year 2- 3.85 (1.26); and Year 3- 3.54 (0.98) Increased implementation of research-based strategies to promote comprehensibility Use of graphic organizers stemming from teacher participation in Thinking Maps Training, Project WRITE, and Project GLAD evident in school sites that reported receiving professional development in these programs Teachers were observed using realia and/or visuals such as pictures, pictorial input charts, diagrams, and overhead transparencies Question and answers were also reported as effective strategies, followed by monitoring comprehensibility through informal assessments by walking around the class, checking students’ papers, or using whiteboards to ask students to show their understanding of a concept or skill Over three-year observation period, an increased number of teachers in the middle and high schools were observed using note-taking and checking strategies to monitor comprehension Primary language use Concerns about when and how to use primary language to clarify learning and check for understanding Sample Anecdotal Notes: “Thinking Maps posted. Teacher generated map and students provided input for map. Students writing sentences based on Thinking Map.” – Elementary school classroom “Pictorial Input charts posted in Spanish and English. Students placed index cards over targeted vocabulary.” – Elementary school classroom “Concept Map used for lesson. Teacher poses question: ¿Qué mas puedes decir de este concepto? (What else can you state about this concept?)” – Middle school classroom

27. Interactions • Theme 4: Predominance of Teacher Directed Instruction - Increased attempts to provide opportunities for varied groupings across k-12th grade classrooms - Many teachers identified this as an area for improvement - Most classroom interactions were teacher centered - Few teachers mentioned language proficiency as factor for grouping - Few teachers provide opportunities for self-selected groups K-12 mean scores and standard deviation reported as Year 1- 3.17(1.07); Year 2- 3.22 (1.08); and Year 3- 3.41(0.99) Most classroom interactions were teacher centered, allowing few opportunities for student-to-student interaction, or even student-to-teacher interaction During three year observations, there were increased attempts to provide opportunities for varied groupings across k-12th grade classrooms, with a clear purpose for tasks and routines for collaboration Few teachers mentioned language grouping and personality and social traits as factors for establishing either homogenous or heterogeneous groups Few teachers stated that they provide opportunities for students to self-selected groups Sample Teacher Interview comments: “In terms of cooperative groups, I have a long way to go.” – Elementary school teacher “We don’t do a lot of group things because there is so much direct teaching to be done.” – Middle school teacher “With this class I found that they don’t work well in groups. I need to figure out whether they need more of that. I haven’t figured out how to best do it. I find that they are not mature enough to do this. I need to build in the culture where they are responsible and working. I’m not sure how to do this yet.” – High school teacher K-12 mean scores and standard deviation reported as Year 1- 3.17(1.07); Year 2- 3.22 (1.08); and Year 3- 3.41(0.99) Most classroom interactions were teacher centered, allowing few opportunities for student-to-student interaction, or even student-to-teacher interaction During three year observations, there were increased attempts to provide opportunities for varied groupings across k-12th grade classrooms, with a clear purpose for tasks and routines for collaboration Few teachers mentioned language grouping and personality and social traits as factors for establishing either homogenous or heterogeneous groups Few teachers stated that they provide opportunities for students to self-selected groups Sample Teacher Interview comments: “In terms of cooperative groups, I have a long way to go.” – Elementary school teacher “We don’t do a lot of group things because there is so much direct teaching to be done.” – Middle school teacher “With this class I found that they don’t work well in groups. I need to figure out whether they need more of that. I haven’t figured out how to best do it. I find that they are not mature enough to do this. I need to build in the culture where they are responsible and working. I’m not sure how to do this yet.” – High school teacher

28. Teacher Professional Development Overall, classroom observations revealed an over-reliance on restrictive curriculum which in turn resulted in a limited use of supplemental materials that are culturally relevant and engaging for students. The most observed method of instruction was teacher directed, with few opportunities for meaning, purposeful learning with varied interactions that allow students to process, internalize, and solidify concepts and skills We observed an increase in targeted efforts to promote comprehensible input and output for maximum student learning; however, these were not often coupled with extensive opportunities for problem solving and critical thinking. Overall, classroom observations revealed an over-reliance on restrictive curriculum which in turn resulted in a limited use of supplemental materials that are culturally relevant and engaging for students. The most observed method of instruction was teacher directed, with few opportunities for meaning, purposeful learning with varied interactions that allow students to process, internalize, and solidify concepts and skills We observed an increase in targeted efforts to promote comprehensible input and output for maximum student learning; however, these were not often coupled with extensive opportunities for problem solving and critical thinking.

29. What impact did PROMISE have on Principals? Franca Dell’Olio, Ed.D Loyola Marymount University Center for Equity for English Learners (CEEL) Research approach: Quantitative and qualitative data were collected from the principals at the 15 school sites (early childhood, elementary, middle and high school). To what extent do principals act upon their current knowledge, skills, & expertise of the PROMISE Core Principles as they relate to ELs? What are the principals’ perceptions of current practices for meeting the needs of ELs? What additional Professional Development is needed? • The Protocol for Advocacy Oriented Leadership & Administration (PAOLA) is a research-based tool for assessing site principal’s perceptions of their own current knowledge, skills, expertise, and implementation of advocacy-oriented leadership as defined by the PROMISE Core Principles: aligned with the California Professional  Standards for Educational Leaders recapitulates six school administrative  domains Structured follow-up interviews and focus groups • Quantitative findings: Analyzed and reported using descriptive statistical measures, specifically, measures of central tendency including arithmetic mean (Chapter 5 - Tables 2 – 7) • Qualitative findings: Analyzed through content analysis approaches, specifically constant comparative method, to generate themes, patterns and trends and reported holistically. (Chapter 5 - Tables 8 – 10) Research approach: Quantitative and qualitative data were collected from the principals at the 15 school sites (early childhood, elementary, middle and high school). To what extent do principals act upon their current knowledge, skills, & expertise of the PROMISE Core Principles as they relate to ELs? What are the principals’ perceptions of current practices for meeting the needs of ELs? What additional Professional Development is needed? • The Protocol for Advocacy Oriented Leadership & Administration (PAOLA) is a research-based tool for assessing site principal’s perceptions of their own current knowledge, skills, expertise, and implementation of advocacy-oriented leadership as defined by the PROMISE Core Principles: aligned with the California Professional  Standards for Educational Leaders recapitulates six school administrative  domains Structured follow-up interviews and focus groups • Quantitative findings: Analyzed and reported using descriptive statistical measures, specifically, measures of central tendency including arithmetic mean (Chapter 5 - Tables 2 – 7) • Qualitative findings: Analyzed through content analysis approaches, specifically constant comparative method, to generate themes, patterns and trends and reported holistically. (Chapter 5 - Tables 8 – 10)

30. By Year 3, Principals reported the following impacts: • PROMISE was a vehicle to stimulate change • Focused site on research-based best practices • Focused site on biliteracy • Built collaboration • Emergence of “new” leaders Impact of PROMISE on Site Administrators

31. Impact (continued): • Identification of program, instruction and materials for Early Education • Elementary Schools - instruction, parental engagement, English Language Development and program consistency • Middle Schools - placement, instruction, parental engagement, and school culture • High Schools - program coherence, appropriate placement, instruction, academic support and monitoring, and student involvement/leadership 

32. Principals reported the following challenges: • Initial misconceptions about PROMISE • The need to create collaborative meaning and group learning • Lack of consistent district level support or vision from district • Establishing and maintaining the PROMISE Lead Teams • Lack of resources 

33. Principals reported the following leadership support & overall professional development as being most useful to them as administrators: • Mid-year symposia and end-of-year reflection and planning sessions • Targeted workshops and trainings • Collaboration with facilitators • Integration of researchers and practitioners afforded by PROMISE partners • Networks formed by PROMISE

34. The PROMISE Initiative and support system provided the opportunity and possibility for much positive growth and gave birth to several notable site administrator best-practices:

35. Impact of PROMISE on Student Achievement Kathryn Lindholm-Leary, Ph.D.

36. • Student-level data – 14,000 EL and R-FEP students in grades 2-12. • Outcomes ? CELDT, CST, CAHSEE, Spanish (Aprenda and STS), and other achievement measures (high school drop out). ? Analyzed by grade level for each PROMISE site and across the PROMISE sites. ? Focus - year 3 and progress over the duration of the   PROMISE Initiative. Data Collected

37. Risk Factors • In comparison to the district, county, and state averages, PROMISE school sites had far more EL, Hispanic, and economically disadvantaged students, and students whose parents had a high school education or less.  • Student risk factors associated with lowered achievement ? Hispanic ? EL ? Economically disadvantaged ? Parent with high school education or less ? Having a disability

38. Results – English Proficiency • Considerable variation across sites in percent of students attaining English proficiency. • Consistency across sites - from grade 7, almost ¾ of students English proficient (R-FEP or Early Advanced/Advanced on CELDT).  • Excellent growth in English language development. • Narrowed gap between the State average and the PROMISE district average.

39. Results – English Proficiency (2009)

40. Results – Achievement in English • PROMISE elementary and middle school students made significant gains in achievement ? Narrowed gap between PROMISE average and State average. • PROMISE high school students showed mixed results: 2/3 passed both the ELA and math sections of the CAHSEE Bimodal patterns - ? declines in scale scores, but increase in the percent of Proficient/Advanced in ELA.  ? pass rates for R-FEP students in PROMISE and the state average were similar for both reading and math.

41. Factors That Influence Achievement in English:Risk Factors • At all grade levels: language proficiency and achievement measures (CST & CAHSEE) highly related to: ? District & School Demographic risk factors – attending higher risk districts and schools lowers student achievement ? Student background/risk factors – more risk factors lower student achievement • R-FEPs scored at least comparable to the State average for R-FEPs and closed the achievement gap with English speaking students in English language arts. 

42. Factors That Influence Achievement in English:Type of Instructional Program

43. Impact of PROMISE on Student Achievement: Conclusions: • Relatively high-risk students in low to high risk settings made good progress toward proficiency and achievement in English. • Considerable variation in student outcomes by sites and grade spans, but ? 3/4 of students proficient in English by grade 7. ? Students made progress in achieving in English language arts and math and passing the CAHSEE. • Participation in two-way programs helped students outperform students in SEI/Mainstream programs.

44. • PROMISE results in EL specific research-based changes • It is a better match for some sites than others, but the model works for all levels of schooling • The PROMISE vision mattered • Core-principles gave coherence to school improvements and led to a more comprehensive reform • Each component of the model is essential to the impact Overall Research Conclusions:

45. • Achievement gains and narrowing of the gap (student impacts) • More research-based practices and models (programs and practices) • Redefined visions, broader ownership for ELs, more distributive and inclusive leadership (culture, leadership) PROMISE resulted in positive impacts at all levels Changes in school structures, program design and policies incorporating more research-based practices Small to moderate increase in use of higher quality classroom practices Principals reported that participation resulted in redefining vision and stronger plans for EL success, more collaborative systems and more school-wide emphasis on ELs.Changes in school structures, program design and policies incorporating more research-based practices Small to moderate increase in use of higher quality classroom practices Principals reported that participation resulted in redefining vision and stronger plans for EL success, more collaborative systems and more school-wide emphasis on ELs.

46. • Opportunity to collaborate was a “draw” • Network increased identifying good practices, and enhanced capacity to replicate • Involving multiple levels of system improved data collection • PROMISE collaboration built relationships that can last beyond the pilot Enabling infrastructure and communities of practice were powerful PROMISE created an infrastructure that enabled the development of communities of practice and networking, technical assistance and professional development with a focus on ELs utilizing existing elements of the school system. Collegial collaboration ranked highly by teachers as factor in improving teaching Engagement of sites, districts, county offices facilitated data collectionPROMISE created an infrastructure that enabled the development of communities of practice and networking, technical assistance and professional development with a focus on ELs utilizing existing elements of the school system. Collegial collaboration ranked highly by teachers as factor in improving teaching Engagement of sites, districts, county offices facilitated data collection

47. • PROMISE ? increased sense of efficacy among teachers and improved practices • More emphasis on longer-term academic impact rather than just short-term accountability demands • More alignment and strengthening of program improvement plans for ELs • Collecting outcomes in Spanish showed evidence of positive outcomes PROMISE assisted responses to challenges of accountability system PROMISE helped educators deal with the challenges in implementing research-based EL approaches in the context of the current accountability system.PROMISE helped educators deal with the challenges in implementing research-based EL approaches in the context of the current accountability system.

48. • Two-way programs outscored SEI/mainstream • Teachers of L1 instruction had higher ratings in effective classroom practices • Deeper understanding of research on L1 led to new and strengthening L1 development programs, purchase of Spanish materials, well-designed programs Biliteracy models improved student outcomes Consistent and articulated biliteracy models have the power to improve student outcomes for English Learners Teachers of L1 instruction had higher ratings in effective classroom practices (esp. rigorous and relevant curriculum)Consistent and articulated biliteracy models have the power to improve student outcomes for English Learners Teachers of L1 instruction had higher ratings in effective classroom practices (esp. rigorous and relevant curriculum)

49. • Schools seeking to improve EL outcomes need support infused with EL expertise • Reform efforts must be appropriately measured and given adequate time • Create partnerships and networks to develop and disseminate models and practices • Long-term and in-depth professional development and leadership development are needed for EL success IMPLICATIONS FOR SCHOOL REFORM

50. • PROMISE is not only a model to initiate and support changes, but is a model for how the school system might function for the ongoing tasks of refining practices and creating schools appropriate for and effective for all students.