The NC Race to the Top Evaluation:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 36

The NC Race to the Top Evaluation: An Update February 14, 2013 PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on
  • Presentation posted in: General

The NC Race to the Top Evaluation: An Update February 14, 2013 Jessica Anderson, SERVE Center, UNCG Julie Marks, CIPP, UNC-CH Trip Stallings, Friday Institute, NCSU. NC Race to the Top Evaluation: 2013 Forward. NC RttT is designed to be a “game changer”

Download Presentation

The NC Race to the Top Evaluation: An Update February 14, 2013

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The nc race to the top evaluation an update february 14 2013

The NC Race to the Top Evaluation:

An Update

February 14, 2013

Jessica Anderson, SERVE Center, UNCG

Julie Marks, CIPP, UNC-CH

Trip Stallings, Friday Institute, NCSU


Nc race to the top evaluation 2013 forward

NCRace to the Top Evaluation: 2013 Forward

NC RttTis designed to be a “game changer”

A coordinated set of innovative activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools

Evaluation contributes to NC RttT “changing the game” in two ways:

  • Program Evaluation: Provide formative information on the implementation of NC RttT initiatives Inform decisions to improve implementation

  • Policy Evaluation: Assess – from the perspective of students, teachers, leaders, and schools – the improvements that have occurred as a result of NC RttT initiatives collectively and individually: Includes LEA-level outcomes for some initiatives Inform decisions about sustainability and impacts

Ending in 2013


Focus of the evaluation

Focus of the Evaluation

  • External evaluation provides objective analysis of the activities described in NC’s RttT grant proposal:

    • Implementation fidelity

    • Short-term outcomes

    • Collective/overall impact

  • Even with the shift from formative to summative, however, this is still notan evaluation of specific teachers, leaders, or schools


Nc race to the top evaluation key milestones in year 2

NC Race to the Top Evaluation: Key Milestones in Year 2

Spring 2012: Completed 2nd administration of the Teacher & Principal Survey (focus shifts to 365 sample schools instead of entire state); completed 3 reports (Comparison of Value-Added models; Regional Leadership Academy cost study; Year 1 STEM report)

Summer 2012: Completed 3 reports (Local strategic staffing overview; Distinguished Leaders in Practice report; initial LEA expenditure report)


Nc race to the top evaluation key milestones in year 21

NC Race to the Top Evaluation: Key Milestones in Year 2

Fall 2012: Completed 4 reports (Technical follow-up to Value-Added report; initial North Carolina Teacher Corps & Teach for America report; Online professional development report; Year 2 STEM report); drafted an overall summary report of findings to date

Winter 2012-13: Completed drafts of 7 reports (Year 2 statewide and Year 1 local professional development reports; Year 1 Regional Leadership Academies activities report; Year 1 New Teacher Support Program report; Year 2 District and School Transformation report; Year 1 Virtual Public Schools blended STEM courses report)


Nc rttt evaluation team

NCRttTEvaluation: Team

Consortium for Educational Research and Evaluation-NC(CERE-NC):

SERVE Center, Carolina Institute for Public Policy, and Friday Institute

Steering Committee: Gary Henry, Terri Shelton, & Glenn Kleiman

Principal Investigator: Gary Henry

Management Committee: Julie Marks, Jessica Anderson, and Trip Stallings

Team Leaders:

Teacher and Leader Effectiveness – Heather Higgens

Equitable Supply and Distribution of Teachers and Leaders – Trip Stallings

Professional Development – Jeni Corn

Turnaround of LEAs and Schools – Charles Thompson

Local-Level Implementation and Spending – Nate Barrett

Overall Impact – Gary Henry and Julie Marks

Other Leadership Roles:

LEA Coordinator – Lynn Amwake ([email protected]; 336 574-8739);

State Liaison - Trip Stallings ([email protected]; 919 513-8576)


Nc rttt initiatives evaluation organization

NC RttT Initiatives: Evaluation Organization

  • Teacher and leader effectiveness

    Integration of value-added student achievement measures into

    educator evaluation system

  • Equitable supply and distribution of teachers and leaders

    Teach for America, NC Teacher Corps, Regional Leadership Academies, Teacher Induction Program, Virtual Public School, Incentives

  • Professional development

    All professional development activities in support of RttT initiatives, including: PD for standards and assessment, IIS, and data use; and PD delivery capacity-building efforts

  • Turnaround of LEAs and schools

    Low-achieving LEAs and schools; STEM schools

  • Local-level implementation and spending on RttT

    Cloud computing, allocation of RttT funds, cost savings

  • Overall impact of RttT on students, teachers, and school leaders

    Cross-Initiative: Omnibus survey of teachers and principals


Key findings 2012 http cerenc org

Key Findings, 2012 (http://cerenc.org)

Local Strategic Staffing – 18 LEAs developed full strategic staffing plans, and 55 others developed partial plans. Highlights: emergence of 2nd-generation strategic staffing plans; potential for sustainability; diversity in strategic staffing approaches; and opportunities for idea-sharing across LEAs.

Distinguished Leadership in Practice – In Year 1, DLP sessions were conducted for 194 principals (157 completed the program). Principal feedback and observation data indicate that face-to-face and online sessions were of high quality. Principal feedback and focus group data suggest that participants developed specific leadership knowledge and skills.


Key findings 2012 http cerenc org1

Key Findings, 2012 (http://cerenc.org)

Online Professional Development – Approximately half of the state’s educators completed at least one online module by the end of the 2011-12 school year. Overall, most participants agreed that the webinars and modules were relevant to their professional development needs, though some content was redundant with prior PD activities and not always content- or grade-specific, and some modules did not meet Learning Forward/NSDC standards. Most online modules were completed independently and not in PLCs.


Key findings 2012 http cerenc org2

Key Findings, 2012 (http://cerenc.org)

LEA Expenditures– Total RttT funds allocated to LEAs averages to $36 per pupil per year, though the range across LEAs is broad ($6-$218). Based on analyses of DSWs, LEAs plan to use the largest proportion of their RttTfunds (49%) for technology, followed by professional development (21%) and strategic staffing (15%). In terms of planned activities, LEAs plan to prioritize RttT funds for professional development (43%) followed by technology (24%) and strategic staffing (20%).

Value-Added Models– Evaluation of nine commonly-used teacher value-added models suggests that, while none of the models performs sufficiently well for high-stakes purposes on its own, four models (a three-level hierarchical linear model with one year of pretest scores, a three-level hierarchical linear model with two years of pretest scores, EVAAS, and a student fixed effects model) are sufficient for lower-stakes purposes.


Key findings 2012 http cerenc org3

Key Findings, 2012 (http://cerenc.org)

NC Teacher Corps & TFA– NCTC recruitment yielded 441 candidates, of whom 34 were selected (29 remain in the program; 22 have teaching jobs). TFA placed 157 corps members in 2011-12, and 219 in 2012-13—many in teaching “pods” of 3 or more. Almost 90% of TFA corps members complete two-year commitments.

STEM Affinity Network: Second-Year Report – Anchor schools have improved instruction and implemented STEM features (such as project-based learning and partnering for improvement of student learning) internally. Structures for networking, PD, curriculum development, and partnerships are in place to support affinity schools, though some of these activities have been delayed. There is not yet universal buy-in among anchor school staff, but many report improved instruction and implementation of STEM strategies. Students in anchor schools enjoy personalized attention and exhibit high motivation, engagement, and a passion for learning.


Omnibus teacher and leader survey

Omnibus Teacher and Leader Survey

  • Administered to probability sample of 365 schools across the state – a selection of schools that is reflective of the state as a whole

  • Assesses “Instructional Climate”

    • 23 dimensions on leadership and organizational conditions affecting instruction

  • Baseline and Second Round Surveys completed – Fall 2011, Spring 2012

  • Next administration in progress soon (launches February 2013) and again in 2014 (last administration)


School level activities reports https s3 amazonaws com evaluation tools evaluationdb index html

School-Level Activities Reports:https://s3.amazonaws.com/evaluation.tools/evaluationdb/index.html


The nc race to the top evaluation an update february 14 2013

Jess Anderson. SERVE Center, [email protected]; (336) 543-6185Julie Marks, CIPP, [email protected]; (919) 962-7422Trip Stallings, Friday Institute, [email protected]; (919) 513-8576


1 teacher and leader effectiveness

1. Teacher and Leader Effectiveness

Purpose of this Evaluation Project

  • To ensure quality, consistency, and fairness of new and ongoing teacher and principal evaluation processes through examination of validity and reliability across multiple observational perspectives

  • To examine educators’ perspectives on new evaluation standards and the effect of these standards on educators’ practices


1 teacher and leader effectiveness1

1. Teacher and Leader Effectiveness

Evaluation Questions

  • Have valid and reliable measures of student growth been identified for inclusion in the teacher and administrator/principal evaluation process?

  • Does the revised evaluation process allow for/make meaningful distinctions between teachers’ and administrators’ effective and ineffective performance?

  • How do educators view the implementation/rollout of the evaluation process? Does the new evaluation process change educators’ attitudes? Does it change educators’ practices?

  • Do performance incentives for teachers in low-performing schools have positive effects on student and teacher outcomes?


1 teacher and leader effectiveness2

1. Teacher and Leader Effectiveness

Evaluation Approaches

  • Data sources: existing scholarship; quantitative data (including longitudinal DPI data and survey data collected from principals, teachers, and students); and qualitative data (including teacher and principal interviews, focus group data, and observations of teachers).

  • Key Deliverables and Estimated Delivery


The nc race to the top evaluation an update february 14 2013

2. Equitable Supply and Distribution of Teachers and Leaders

Purpose of this Evaluation Project

  • To provide both summative and formative information about RttT efforts to increase the overall supply and to ensure the equitable distribution of effective educators statewide

Evaluation Strands

  • Baseline

  • Regional Leadership Academies

  • NCTC & TFA Expansion

  • Strategic Staffing

  • New Teacher Induction

  • NCVPS Blended STEM Courses


2 equitable supply and distribution of teachers and leaders

2. Equitable Supply and Distribution of Teachers and Leaders

  • Overall Evaluation Questions

  • What is the nature and quality of the experiences provided by each of the initiative programs?

  • Are students affected by each of these programs better off than students in schools and districts not served by these programs?

  • Are these initiatives cost-effective and sustainable?

  • To what extent did the initiatives further the goal of having an effective teacher in every classroom and an effective principal in every school?


2 equitable supply and distribution of teachers and leaders1

2. Equitable Supply and Distribution of Teachers and Leaders

Evaluation Approaches: Not just . . . .


2 equitable supply and distribution of teachers and leaders2

2. Equitable Supply and Distribution of Teachers and Leaders

Evaluation Approaches: Also . . .


2 equitable supply and distribution of teachers and leaders3

2. Equitable Supply and Distribution of Teachers and Leaders

Key Deliverables and Estimated Delivery (I)


2 equitable supply and distribution of teachers and leaders4

2. Equitable Supply and Distribution of Teachers and Leaders

Key Deliverables and Estimated Delivery (II)


3 professional development

3. Professional Development

Purpose of this Evaluation Project

  • To conduct ongoing analysis of the delivery and quality of state- and local-level professional development, with the goal of analyzing the impact of the PDI on local capacity, teacher practices, and student achievement.

  • We will examine longitudinal education data combined with data collected using a sample of schoolsapproach.  


3 professional development1

3. Professional Development

Major Evaluation Questions

  • State Strategies: To what extent did the state implement and support proposed RttT PD efforts?

  • Short-Term Outcomes: What were direct outcomes of State-level RttT PD Efforts?

  • Intermediate Outcome: To what extent did RttT PD efforts successfully update the NC Education Workforce?

  • Impacts on student performance: To what extent are gains in student performance outcomes associated  with RttT PD?


3 professional development2

3. Professional Development

Evaluation Approaches

  • Interviews and focus groups

  • Observations PD, teaching practices, PLCs, local PD Sessions

  • Research-based review of PD content and delivery, including reviews of the NCDPI Online Repository and eLearning Portal diagnostics

  • Data reviews and analyses administrative data, PDI-specific data (PDI Participation Database data, survey data including PD Exit Surveys, leadership inventory, reflection), NCEES summary data, student data including EOG/EOC, graduation rates; LEA PD expenditure data

  • Document reviews LEA PD Action Plans, other PDI-specific documents

    Key Deliverables and Estimated Delivery


4 turnaround of leas and schools

4. Turnaround of LEAs and Schools

Purpose of this Evaluation Project

  • To understand the extent to which and the ways in which interventions by the District and School Transformation division (DST) improve outcomes for students in the state’s lowest-performing schools and districts

  • To explore the fidelity of implementation of the STEM Schools initiative and examine its impacts on students, teachers, principals, schools, and school networks.


4 turnaround of leas and schools1

4. Turnaround of LEAs and Schools

Turnaround of LEAs and Schools Evaluation Questions

  • What problems are identified in the low-performing schools and districts?

  • What are the main intervention strategies that the District and School Transformation unit employs to improve low-performing schools?

  • What are the intended mechanisms of improvement?

  • How do the strategies work? Do the strategies and mechanisms play out as intended?

  • What is the impact of the intervention strategies on intermediate outcomes as well as student achievement and graduation rates?


4 turnaround of leas and schools2

4. Turnaround of LEAs and Schools

STEM Anchor Evaluation Questions

  • To what extent have the network of STEM anchor and cluster schools been implemented as intended?

  • What are the impacts of the network of STEM anchor and cluster schools on student and on school-level outcomes and how do these impacts compare with the impacts of other transformation models?

  • Can the impacts on student performance be disaggregated by student and school characteristics?

  • What mechanisms are put in place for the sustainability and scaling up of the model, or its most successful elements?


4 turnaround of leas and schools3

4. Turnaround of LEAs and Schools

Evaluation Approaches

  • DST: 30+ site visits, multiple interviews (over the next three years with multiple people), observations, focus groups, surveys, artifact review, document analysis, and quantitative analyses.

  • STEM: Predominantly qualitative analyses (observations of professional development, site visits to STEM schools, interviews with providers), with quantitative analyses (student and school staff surveys, administrative data).

    Key Deliverables and Estimated Delivery


5 local level implementation spending

5. Local-Level Implementation & Spending

Purpose of this Evaluation Project

  • To determine how Race to the Top funding is being allocated and used across districts and schools throughout NC


5 local level implementation spending1

5. Local-Level Implementation & Spending

Cloud Questions

  • To what extent does the Cloud reduce state & local expenditures for technology?

  • To what extent does the Cloud provide reliable, secure, accessible, and efficient service?

  • How satisfied are LEAs with the Cloud Computing infrastructure?

    Local Spending Questions

  • How do local districts spend RttT funds?

  • Are some local RttT spending patterns associated with higher student performance in schools and districts?

    Local Efficiencies and Savings Questions

  • Do RttT funds alter costs incurred by the state and districts?


5 local level implementation spending2

5. Local-Level Implementation & Spending

Evaluation Approaches

  • Mixed-method approach, combining document review, interviews, surveys, and quantitative analysis of administrative data

    Key Deliverables and Estimated Delivery


6 overall impact

6. Overall Impact

Purpose of this Evaluation Project

  • To provide estimations of the overall impact of RttT-funded initiatives

  • To explore under what conditions and circumstances the initiatives collectively and in various combinations appeared to be most effective, and for whom

  • To consider sustainability options beyond the life of the grant

  • To track and compare the metrics/goals defined in the proposal


6 overall impact1

6. Overall Impact

Evaluation Questions

  • Was each RttT initiative implemented as intended?

  • What are the overall impacts of RttT on increasing student performance, such as achievement, engagement, attendance, graduation?

  • Are the impacts of RttT on student performance larger in some schools/districts than others (for example, high-poverty or low-performing schools)?

  • Are some RttT initiatives more effective in increasing student performance than others?

  • How can the successful RttT initiatives be sustained after 2014?


6 overall impact2

6. Overall Impact

Evaluation Approaches

  • The Overall Impact evaluation will consider selected quantitative and qualitative data and results from all initiative-level evaluations, as well as cross-initiative and all-inclusive data

    Key Deliverables and Estimated Delivery


  • Login