Assessment Update

1. Assessment Update FOIL November 16, 2011 Dr. Sharon Koon Assistant Deputy Commissioner Accountability, Research, & Measurement Florida Department of Education

2. Topics FCAT 2.0/EOC Assessment Standard Setting Update 2012 Assessment Data Forensics

3. Low High Level 1 Level 2 Level 3 Level 4 Level5 Standard Setting: FCAT 2.0 Reading, Mathematics, and Algebra 1 EOC Assessment There are five Achievement Levels Requires the setting of four Achievement Level cuts Five Achievement Levels, Four Cut Points

4. Standard-Setting Process and Timeline

5. Outcomes from the Educator Panels September 20-23, 2011

6. Scale Scores – Reading, Educator Panel

7. Impact Data – Reading, Educator Panel

8. Scale Scores – Mathematics, Educator Panel

9. Scale Scores – Algebra 1 EOC Assessment, Educator Panel Recommended College-Ready Cut Score – Achievement Level 3 Algebra 1 EOC Assessment scale score range: 325 to 475.

10. Impact Data – Mathematics, Educator Panel

11. Reactor Panel Review • Considered the following: • Information and materials from the standard-setting meeting, including variability of educator judgments • Achievement Level Policy Definitions and Descriptions • External tests that are commonly administered to Florida students outside of the FCAT 2.0 and Algebra 1 EOC Assessment system • NAEP, Stanford 10, PSAT, SAT, PLAN, ACT, College-Readiness Data (CPT, ACT, SAT) • Impact data • By subject/grade • By gender • By ethnicity • Across subjects and grades (“vertical articulation”)

12. Key Review Questions • Reactor panel considered the following questions: • Do the impact data for each grade look reasonable compared to other grades? Considered that AL 3 is now the graduation standard. • Is this the expected pattern of impact data across grades and between subjects? • How does the impact data compare to external data? • Should the cut scores be moved higher (higher expectation) or lower (lower expectation)? Why?

13. Impact Data – Reading, Reactor Panel

14. Scale Scores – Mathematics, Reactor Panel

15. Impact Data – Mathematics, Reactor Panel

16. Draft Proposed Rule Posted online for public input

17. Draft Proposed Rule • Changes to Reactor Panel Outcomes: • Set the bar higher for Achievement Level 5 – by setting the expectation such that no more than 10% of 2011 students would have achieved Level 5 in all grades and subjects. • Made a small correction to the required scale score to achieve Level 4 in Grade 8 Reading.  • It was slightly inconsistent with the other grades (slightly more rigorous) and subtracting one scale score point (250 to 249) achieved consistency.

18. Scale Scores – Reading, Draft Proposed Rule Not determined

19. Impact Data – Reading, Draft Proposed Rule Not determined

20. Scale Scores – Mathematics, Draft Proposed Rule

21. Scale Scores – Algebra 1 EOC Assessment, Draft Proposed Rule Recommended College-Ready Cut Score – Achievement Level 3

22. Impact Data – Mathematics, Draft Proposed Rule

23. Final Steps Notice of Change to be published in FAW, with the Commissioner’s recommended cuts scores for FCAT 2.0 Reading, grades 8-10 State Board of Education Decision – December 5, 2011. Rule Adoption

24. Spring 2012 Data Forensics Purpose Legal Defensibility Plans Resources

25. Purpose: Responsibilities of the Testing Program • Ensure the test scores represent the student’s knowledge or ability • Protect the integrity of the assessments • Focus on the scores … not on behavior, not morals

26. Legal Defensibility • Is cheating a crime? • For adults – yes! Section 1008.24, F.S. Misdemeanor of the 1st degree. • Section 1008.24, F.S, does not address student answer copying. • Cheating is sometimes difficult to prove • Defensibility standard: “Act in Good Faith”

27. Acting in Good Faith • Clear communication of policy • Fair treatment of test takers • Demonstrate due process • Justify policy decisions • Respect test takers’ rights • Protect integrity of the assessments and the student’s educational opportunities • Courts show great deference to testing programs when they have acted in good faith

28. What are we acting on?Example: Similarity Index ≥ 12 • Student response patterns being so similar that the probability of seeing two (or more) students’ tests so similar, with each doing his/her own work: 0.000000000001 • Does not equal cheating, means test score is not trustworthy • Fairness and validity of test instance must be questioned

29. Example: Median Similarity, Index = .3

30. Example: Extreme Similarity, Index = 15.8

31. Example: A Comparison…

32. Important Points to Remember this Spring • Security breaches are real • We can tell a lot about the strength of security from the data • Security is best strengthened before the breach occurs • Legal defensibility is based on “good faith” • Positive results follow consistent application of security principles

33. Plans: Spring 2012 Data Forensics Students with extremely similar test responses Schools with improbable similarity, gains, or erasures

34. Student-level Analyses • The Department will continue to flag and invalidate the test scores of students in the same school whose test responses were so similar and statistically aberrant that the validity of the test responses must be questioned. • A conservative threshold of 1 in 1012, or one in a trillion, will continue to be used in flagging scores.

35. Appeals Process The Department will continue to provide an appeals process for districts to present evidence describing the testing circumstances that explain why invalidated results should be considered valid. Districts should notify schools and parents that an appeal process is available.

36. School-level Analyses • Detection of extreme levels of similarities, gains, and/or erasures at the individual school level. • The Department will require the district to conduct an internal investigation to identify reasons for the extreme results. • The Department will review the results for each district investigation. • If an investigation is not resolved prior to the release of school grades, the grade for that school will be held until a resolution is reached. • Unsettled cases may result in the involvement of the Department’s IG’s office, Professional Practices, as well as the FDLE.

37. Resources … coming soon! • A reference guide is being produced by the Department to assist districts in the process of appealing test score invalidations. The appeals guide will: • outline the materials and information necessary or helpful to submit with an appeal, such as seating charts, student statements, and test administrator statements. • describe basic procedures districts might follow in conducting investigations.

38. Resources …coming soon! • Parent/Guardian Letter ~ revised letter for use in notifying a parent/guardian that his or her student’s score has been invalidated. The letter will describe the data forensics process in laymen’s terms. • Data Forensics Results Training ~ web-based presentations. • Wednesday, November 30, from 10:00-11:30 a.m., or • Thursday, December 1, from 2:30-4:00 p.m. • Data Forensics Technical Information ~ will be provided in February 2012.