1 / 40

Fast Track Program Evaluation Using Assessments Diagnostically

Fast Track Program Evaluation Using Assessments Diagnostically. Philip A. Streifer, Ph.D. Bristol Superintendent of Schools; UCONN Executive Leadership Program Yvel Crevecoeur Doctoral Student, Department of Educational Psychology, program in Special Education. Primary Issues.

feo
Download Presentation

Fast Track Program Evaluation Using Assessments Diagnostically

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Fast Track Program Evaluation Using Assessments Diagnostically Philip A. Streifer, Ph.D. Bristol Superintendent of Schools; UCONN Executive Leadership Program Yvel Crevecoeur Doctoral Student, Department of Educational Psychology, program in Special Education

  2. Primary Issues • Bristol an urban leader in reform • Next phase of development: program evaluation and focus on instructional excellence • Primary reading intervention program: Fundamentals of Literacy Development (FOLD) • Rising CT AYP Standard for 2008: Need to track student gains over time vs. NCLB/AYP year by year analysis

  3. Pressures • 181 First Days of School • Growing Poverty Rate • Manage Expectations in City • Article for upcoming ‘Chamber Views’ • Ongoing speaking and publications on challenges • Focusing work of 1000 people • Rising AYP Standards and City confidence in schools: can we ‘get over the bar’?

  4. Challenges • Migration • Economically Disadvantaged • Rising State NCLB Standards for 2008 • Reading – 68 to 79% proficiency • Math – 74 to 82% proficiency • Special Education Placements and Costs

  5. 181 First Days of School One School’s Experience Transfers In Transfers Out July-Aug- Sep 61 October 8 November 14 December 3 January 8 February 12 March 10 April 2 May 3 June 1 TOTAL 122 • July-Aug- Sep 53 • October 16 • November 11 • December 4 • January 24 • February 12 • March 11 • April 1 • May 0 • June 0 • TOTAL 132 GRAND TOTAL for ONE YEAR: 254 Children in and out/School Population +/-350

  6. Data Driven Reform and Improvement Past, Present and Future

  7. DISTRICT DATA TEAM POWER STANDARDS SCHOOL DATA TEAM UNWRAPPED STANDARDS IMPROVED STUDENT ACHIEVEMENT BIG IDEAS ESSENTIAL QUESTIONS GRADE/DEPT DATA TEAM CURRICULUM & PACING GUIDES ASSESSMENT OF STUDENT LEARNING EFFECTIVE TEACHING STRATEGIES The Bristol Accountability Initiative Data-Driven Decision Making MAKING STANDARDS WORK

  8. CT Mastery Test and CAPT • CMT – new version for 2005-06 • CAPT – new version for 2006-07 • Both are tougher tests • Each year NCLB/AYP standard rises • NCLB/AYP standard for 2007-08 (spring 2008) is rising dramatically

  9. Matched Cohort Data (2006 to 2007) Writing*

  10. District CAPT 2007 Results Areas highlighted in yellow show where Bristol exceeded State averages.

  11. Performance Level Graph – District vs. State Averages

  12. Gains Analysis: Matched Cohorts • Grade 3 to 4 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Expected Gain • Grade 4 to 5 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Loss Below Expected • Writing: Expected Gain

  13. Gains Analysis: Matched Cohorts • Grade 5 to 6 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Expected Gain • Grade 6 to 7 (2006 to 2007) • Math: Expected Gain • Reading: Expected Gain • Writing: Expected Gain

  14. Gains Analysis: Matched Cohorts • Grade 7 to 8 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Moderate Bain Beyond Expected • Grade 8 to 10 (2004 to 2007) • Math: Moderate Gain Beyond Expected • Reading: Expected Gain • Writing: Expected Gain

  15. Role of Program Evaluation • FOLD: • working; needs to be expanded • Read-180 vs. ReadAbout: • Gains Analysis • ReadAbout Deployed in More Classes 2007-08 with Grant Funds

  16. Return on Investment • Bristol Per Pupil Expenditure is 125th out of 169 (upper end of lower third) • CMT 2007: at or just above state averages • CAPT 2007: above state average • City of Bristol getting a good return on its educational investment

  17. New Pressing Issue • CT AYP standard rises dramatically in 2008 in reading and math • Projection of 2008 AYP indicates many schools will be cited under NCLB • Current district performance at state average on CMT; above state average on CAPT • Poor AYP performance in 2008 will likely cause erosion of public confidence and City support • Deeper analysis ongoing to determine: • Areas of focus for each school • Safe Harbor status and why it is important • Gains analysis to show progress of students over time who remain in the district; to shift focus and opinion and show staff that they are doing a good job

  18. Key Strategies Going Forward • AYP and Safe Harbor – achieving improvement for all children • Program Evaluation – FOLD (first pilot) • Gains Analysis – Cohort Improvement Over Time and Across Disparate Tests • Instructional Excellence

  19. Preliminary 2008 AYP Analysis

  20. FOLD Evaluation • Why were Hubbell Elementary School’s reading scores so high? • Development of the Foundations of Literacy Development (FOLD) evaluation process • Identifying Growth Model of Improvement for each elementary school • “SWAT Team” and focused plans on other intervnetions • Long term objective: Maximizing what really works for students participating in FOLD.

  21. FOLD Evaluation Process • Semi-structured interview of principal who originally implemented FOLD: • Key design features • Developed pilot survey questions • Reviewed and refined survey questions • Administered survey to literacy teachers and principals

  22. FOLD Evaluation Process • Preliminary method of analysis: • Identified each school’s performance on indicators of DRA2, CMT, and history of achievement • Ranked each school’s performance to identify: • Differences across high and low performing schools • Differences between respondents (i.e., principals and literacy teachers) to identify features of implementation that need revision

  23. Gains Analysis Logic • AERA paper presentation 2007 • Data must be matched pairs and equal interval scales • Rescale data if using different tests • Must know the absolute possible range of each dataset • Apply Modified Effect Size analysis • Interpret with reconceptualization of Cohen’s d

  24. Gains Analysis – Different Scales • Rescale • Reset to zero scale • Ratio of one scale to the other • Analysis • Cohen’s denominator for pooled SD for populations (for unequal variances

  25. Interpretation: Less Overlap = More Difference in Scores +/- We would expect little overlap and a positive Cohen’s d

  26. Transformation Logic – Step 1 • Two Scales: • 0 to 10: range = 10 • 20 to 40: range = 20 • Step 1: Put scales on same starting point: • 0 to 10: range = 10 • 0 to 20: range = 20 • [Scores are reset to start at a value of zero. That is, tests with possible scoring ranges of A→B are rescaled such that scores range from 0→(B-A).]

  27. R1 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

  28. R1 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ratio of R1 to R2 = .5

  29. R1 | | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | 10| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Step 2: Multiply R2 scores by ratio of the scales (.5) to reset to R1 scale

  30. R1 | | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | 5 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

  31. Rescale • Range1/Range2 = Resets one score to the other • Example: • R1/R2 = 5/10 = 0.5(or…R2/R1 = 10/5 = 2.0)

  32. Reconceptualizing Cohen’s d in this Context

  33. Gains Analysis Findings…So Far All Schools Improving the Same Over Time: Poorer Performing Schools Not Catching Up – Need Faster Growth – May be Unreasonable to Achieve

  34. Using Data Mining to go Deeper - Variables • School • Grade 3 Teacher • Grade 4 Teacher • Grade 5 Teacher • Gender • Ethnicity • SPED status • F-R Eligible • ELL status • GR5 CMT Math Scale • GR5 CMT Read Scale • GR5 CMT Writing Scale • GR4 CMT Math Scale • GR4 CMT Read Scale • GR4 CMT Write Scale • GR3 DRA Reading Level

  35. Results So Far: • Poverty not a significant factor in achievement; all schools performing: one year gain (+/-) for one year instruction • Teacher is associated with achievement – but not clear as to how • Strong teachers are assigned in many cases to lower performing students which probably accounts for gains analysis • Need to focus interventions on specific groups of students; identified through data mining and ‘teacher/administrator knowledge’

  36. Summary • Past work by district on DDDM and Cultural acceptance critical • Next Steps: • Program evaluation and advanced data analysis • Focus on instructional excellence • Marzano: Effective Teaching Strategies • Saphier & Gower: The Skillful Teacher • Maintain City confidence – get to ‘safe harbor’ for 2008 • Change the Conversation from NCLB/AYP to Growth Over Time

More Related