Use of Confidence Intervals in Performance Improvement Appeals

1. Use of Confidence Intervals in Performance Improvement Appeals Performance Reporting Division Texas Education Agency

2. Overview of CI Process • Eligibility for application of CI • Matched pairs of students • Recalculate performance improvement • Recalculate required improvement • Calculate upper limit of CI • Compare upper limit to recalculated RI

3. Eligibility for Application of CI • To be eligible to have confidence intervals applied to performance improvement on appeal, the district/campus must have shown improvement on: • The measure in question, and • The other measure (graduation or attendance for the specific student group). • Appeals for CI by ineligible districts/campuses were denied.

4. Matched Pairs of Students • Next we used the student level data to create matched pairs of students. • Students in 2004 were matched by student group and test (TAKS vs. SDAA, etc.) to students in 2005. • Unmatched students were dropped from the analysis.

5. Recalculate Performance Rates • Passing rate of the matched pairs of students in 2004 (ppy). • Passing rate of the matched pairs in 2005 (pcy). • A third passing rate, the passing rate of matched pairs across both years (ppc) was also calculated, which will come into play later.

6. Recalculate Performance Improvement • Performance improvement was calculated as the difference between the passing rate of the matched pairs of students in 2004 (ppy) and the passing rate of the matched pairs in 2005 (pcy).

7. Recalculate Required Improvement • Required improvement was calculated by plugging ppy (matched pair performance in 2004) into the RI formula on p. 26 of the 2005 AYP Guide. RI = 100 – ppy 10

8. Calculate Confidence Interval • A 68% one-tailed (upper limit only) Wald confidence interval for performance improvement was computed. UL = PI + [(z/sqrt(n))*sqrt((ppy*(1-ppy))+(pcy*(1-pcy))+(2*((ppy*pcy)-ppc))] Where: UL = Upper limit of the Wald confidence interval PI = recalculated performance improvement rate (pcy - ppy) z = z-score for selected confidence level (for a 68% CI, z = 0.47) n = number of matched pairs ppy = performance (passing rate) of matched pairs in prior year pcy = performance (passing rate) of matched pairs in current year ppc = performance (passing rate) of matched pairs in prior year and current year

9. We generate a range of potential PI values based on the recalculated PI and its standard error to get a normal curve.

10. Calculating the upper limit using the Wald formula gives us a range (or interval) of PI values we are confident the true PI value falls in (the green area). Upper Limit

11. Compare Upper Limit to RI • The upper limit of the confidence interval must be greater than or equal to required improvement for the appeal to be granted. • Another way to say the above is that required improvement must fall inside the confidence interval of performance improvement for the appeal to be granted.

12. Compare Upper Limit to RI If RI falls within CI (green area) then appeal for CI is granted! If RI falls outside the CI, then appeal is denied. NOTE: Because RI will never be less than zero, the real area of interest is the portion of the green area between 0 and the UL.

13. 2005 Preliminary AYP (Reading, Special Education group) 2005 Met Standard = 49% 2004 Met Standard = 44% Change (PI) = 5% RI = 6% 0.1% improvement on other measure FAILS SAFE HARBOR Example from an Actual Appeal

14. Example from an Actual Appeal (cont.) On Appeal (Matched pair data) • Pcy = 42% • Ppy = 22% • Ppc = 12% • Recalc. PI = 20% • Recalc. RI = 8% • 68% Wald UL = 24% GRANTED NOW MEETS AYP