FCAT Writing+ Scoring/Reporting Issues 2007 Annual Assessment Coordinators Meeting

1. FCAT Writing+Scoring/Reporting Issues2007 Annual Assessment Coordinators Meeting Cornelia S. Orr, Assistant Deputy Commissioner of Accountability, Research, and Measurement (ARM) Office of Assessment and School Performance Florida Department of Education “Accept the challenges, so that you may feel the exhilaration of victory.” General George S. Patton

2. Topics • Timeline for Writing+ • Composite Scores Reported • 2007 unscorables • 2008 Grade 10 Change Assessment Coordinators Meeting

3. FCAT Writing+ Measures the Sunshine State Standards in Writing 1999 2006 10th Grade Writing Graduation Requirement in Law 2008 Scores reported on Writing+ Grade 10 students receive scores that count for graduation 2010 2007 1993 2005 First class with Writing+ Graduation requirement Report Achievement Levels State test of writing skills using essay Multiple-choice items added to essay Assessment Coordinators Meeting

4. The Essay Portion • “On demand” draft writing in a 45 minute time period • No knowledge of topic in advance • Planning sheet and two lined pages for draft • Grade 4: Narrative and Expository Modes • Grade 8 and 10: Expository and Persuasive Modes • Scored by trained readers on a scale of 1 to 6 Assessment Coordinators Meeting

5. FCAT Writing+ Scores • Total scale score between 100 and 500 • Essay score (a rubric score of 1 to 6) • Multiple-choice scores for four elements of writing (points correct out of points possible) • Focus • Organization • Support • Conventions • Total scale score is a Composite Score (Essay score + Multiple-Choice score) Assessment Coordinators Meeting

6. FCAT Writing+ Composite Scores Assessment Coordinators Meeting

7. 2007 Writing Issue • The number of unscorable expository essays doubled • Examination of the Issue • Reviewed historical data • Reviewed 2007 process • Reviewed 2007 papers Assessment Coordinators Meeting

8. Scoring Process • Answer documents are scanned • Images created and displayed on computer monitors – Several enhancements over paper • enlargement, brightness, & contrast • Process of Assigning Unscorable Codes • Scorers can assign only Blanks=A • Supervisors and DOE staff on site assign other unscorable codes • Unclear papers can be pulled • No change in the process from previous years Assessment Coordinators Meeting

9. What is Unscorable? • Do not score codes – local decisions • Blank documents (Code=A) • Unscorable Ratings (assigned only by DOE or scoring supervisor) • Off topic (Code=B) • Illegible, Incomprehensible, or Insufficient (Code=C) • Foreign language (Code D) Assessment Coordinators Meeting

10. 2007 Increase in Unscorables • Grade 4 • Primary Reason: Illegible • Increase over 2006: Illegible on Expository • Grade 8 • Primary Reason: Illegible • Increase over 2006: Off Topic on Expository • Grade 10 • Primary Reason: Illegible • Increase over 2006: Illegible on Expository Assessment Coordinators Meeting

11. Comparable Data – Other Years • Grade 4 – 0.46% to 1.01% • 2002 Exp=1.01%; 2002 Nar.=0.94% • 2007: Exp=0.91%; Nar.=0.49% • Grade 8 – 0.22% to 54% • 2004 Exp=0.52%; 2002 Per=0.54% • 2007: Exp=0.54%; Per=0.46% • Grade 10 – 0.16% to 0.80% • 2005 Exp=0.53%; 2000 Per=0.56% • 2007: Exp=0.80%; Per=0.77% Assessment Coordinators Meeting

12. Grade 10 Spring 2008 Change • Only 1 prompt • Randomly selected mode • Expository • Persuasive • Retakes – only one prompt • No change for grade 4 and 8 Assessment Coordinators Meeting

13. Assessment Issues 2007 Annual Assessment Coordinators Meeting Cornelia S. Orr, Assistant Deputy Commissioner of Accountability, Research, and Measurement (ARM) Office of Assessment and School Performance Florida Department of Education “Accept the challenges, so that you may feel the exhilaration of victory.” General George S. Patton

14. Topics • Grade 3 Scoring Anomaly • What happened? • FCAT External Review • Budget Cuts Assessment Coordinators Meeting

15. The Grade 3 Test in 2006 • Passages – Questions – Forms • Student scores based on 5 passages & 45 questions • 30 different forms, each with 1 passage & 7-8 questions • Forms are used for anchor and field test questions • One of the 6 passage positions is used for anchor and field test questions Assessment Coordinators Meeting

16. The Grade 3 Test in 2006 Assessment Coordinators Meeting

17. “Test Construction” • Process of building the test • Occurs the summer before a test • Based on available passages, questions, and statistics • Guidelines for building the test • Test Construction Specifications • Building the test is an iterative process Assessment Coordinators Meeting

18. Equating Solutions Issues • The 2006 equating solution was affected by location changes of the anchor questions • Some change in question location is expected • In 2006, a large number of questions changed locations • 12/21 anchor questions moved more than 15 item positions (0 in 2004 and 2005, and 2 in 2007) • The result was better performance • However, no outliers were identified via calibration • Goal is to find a “better” or “best” equating solution • “Better” must be defined • Process considerations • Selection of anchor questions • Follow the guidelines • Evaluate the quality of the anchor questions Assessment Coordinators Meeting

19. FCAT External Review Process • Advisory Committee • Superintendents and Assessment Coordinators • External Groups – FCAR, Unions, Business Representative • Ad hoc technical subcommittee • Identify and select Independent Reviewers • Identify other technical topics for the full committee • Next Meetings • Thursday, 5:30 • October 2 – Royal Caribe Hotel, Orlando Assessment Coordinators Meeting

20. Independent Review Contract • Review of Experts (individuals and organizations) • Request for Application (non-profits or university affiliates) • Two respondents – Buros Institute (selected) and EduMetrics • Scope of Work • Independent review of 2008 tests • Review of Florida equating procedures • Review/Recommendations about 2006 Grade 3 analyses • Concurrent scoring oversight • Other issues - TBD Assessment Coordinators Meeting

21. Changes Proposed for 2008 • Strict limits on passage/item movements • P-value matched between scored test and anchor items • All equating completed before any grade level is reported • Grade 3 results delayed • Concurrent quality control of equating by independent reviewers • May delay reporting of all grades Assessment Coordinators Meeting

22. Budget Cuts – Impacts on Assessment • Meeting Participants • Increase in refusals – HELP! • Support for substitutes in Federal $ for professional development • Other Scenarios • Reduce printed products • Stop the FCAT Parent Network • Delay the NRT implementation timeline • Writing+ graduation requirement • Reduce FTCE administration and revision Assessment Coordinators Meeting