Redesign of Alternate Assessments for Students with Significant Disabilities: National and State Perspectives

1. Redesign of Alternate Assessments for Students with Significant Disabilities: National and State Perspectives CCSSO June 16, 2008 Redesign of Alternate Assessments

2. Presentation Team • Moderator: Patty McDivitt, Data Recognition Corporation • Aran Felix, Alaska Department of Education and Early Development • Melissa Fincher, Georgia Department of Education • Claudia Davis, Louisiana Department of Education • Discussant: Rachael Quenemoen, National Center for Educational Outcomes Redesign of Alternate Assessments

3. Presentation Focus • Purpose and Rationale • Goals • Development • Challenges • Lessons Learned Redesign of Alternate Assessments

4. Alaska • Aran Felix Alaska Department of Education and Early Development Redesign of Alternate Assessments

5. Purpose and Rationale • Alaska’s previous Alternate Assessment • Portfolio assessment • Housed in the Special Education Unit • Assessed English/Language Arts, Math, and Skills for Healthy Living (only E/LA and Math used for AYP) • Intended use for grades 3, 6, 8, 11 only • Stressed academic content • Alternate Performance Standards (content standards) • Used as Goals on student IEPs • Stressed inclusion (Generalization dimension of scoring) • Department of Education organized scoring sessions • Single set of achievement standards • Scoring dimensions: Skill (student achievement), Generalization, Appropriateness Redesign of Alternate Assessments

6. Purpose and Rationale • Steps to explore a solution: • Alternate Assessment moved to Assessment Unit • Teacher and Parent Survey conducted • Reliability-Validity Study conducted • Considered removing Generalization & Appropriateness dimensions from scoring for AYP; report only to districts. • Needed a new standard setting and better overall technical quality • OSEP Condition drove development of a Data-folio assessment for grades 4, 5, 7, 9, 10, Portfolio for all grades overwhelming teachers. • Accomplished the original intent of inclusion and teaching/assessing content standards • Moved scoring to test vendor • Passed Peer Review requirements Redesign of Alternate Assessments

7. Goals • Assess a broader range of content standards • Manage test size and testing window • Increase standardization • content assessed (apples to apples) • training of teachers to administer the assessment • scoring of the assessment • training of the protégés • Establish a higher technical quality • Create an online assessment (training, scoring, reporting) Redesign of Alternate Assessments

8. Development 1) Funding Issues • Small population of alternate-eligible students • Request for Information 2) What type of assessment to develop? • Research other states’ approaches • Research using another state’s item bank • Decision needed: Retrofit portfolio with performance tasks or move to a performance task system? Redesign of Alternate Assessments

9. Development • Look of new assessment • Used another state’s item bank while examining alignment the first year • Performance Tasks in content areas • 4-8 items per task • Paper/pencil materials allowing for some adaptation • Assessors administer one-on-one to student • Online test materials (scoring protocols, student materials, training manuals) • Online training for test administrators plus proficiency modules to maintain reliability • Online scoring, reporting, unofficial report • Original 3-year plan compressed to 2 years (peer review) Redesign of Alternate Assessments

10. Development • Year 1: 2005-2006 (Portfolio still in place) • Developed Request for Proposals • Contracted with new vendor • Convened committees of educators • Developed Proficiency level descriptors and • Developed Extended Grade Level Expectations • Piloted online system with technology coordinators • Created security access levels • Developed online security rules • Trained Qualified Assessors for Pilot • Pilot tested the new assessment • Created a crosswalk of test items to Alaska content standards (phase 1 blueprint) Redesign of Alternate Assessments

11. Development • Year 2: 2006-2007 • Developed cousin items • Conducted bias review • Conducted statewide trainings and orientations • Developed a qualified assessor and qualified mentor-trainer path and materials • Included mentor pre-test and debrief audios • Administered assessment (one test) • Piloted science assessment • Conducted standard setting for RWM • Received Technical Report • Submitted new assessment to Peer Review (and received approval) • AYP and student reports to parents/districts/website Redesign of Alternate Assessments

12. Development • Year 3: 2007-2008 • Developed cousin items and conducted bias review • Repackage all content tests into grade clusters vs. grade level within content areas • New look for website • Trained Mentors on science assessment • Administered all assessments including science • Teacher Survey of Consequential Validity, Mentor Audios • Conducted standard setting for science • Conducted standard validation for • Received Technical Report • Submitted new assessment to Peer Review (and received approval) • Piloted science assessment • AYP and student reports to parents/districts/website Redesign of Alternate Assessments

13. Development • Future Plans for years 2008-2011 • Develop larger item bank, conduct reviews • Field test new items • Construct new operational test forms, A & B • Conduct a new Standard Setting on Forms A and B of each content-area assessment • Consider a program review by external evaluator Redesign of Alternate Assessments

14. Challenges • Compressing the schedule from 3-4 years to 2 years • Possibility of administering two Alternates simultaneously • Using another state’s items • “Alaskanizing” the assessment • Moving from a one-size model to a grade-cluster model Redesign of Alternate Assessments

15. Lessons Learned • The importance of flexibility and responsiveness • Understanding capacity when huge demands (such as compressed time schedule) are imposed by second peer review deadlines • The role of an outside consultant as EED advisor for the first years of the project • Involvement of TAC prior to issuing RFP Redesign of Alternate Assessments

16. Recommendations • RFP: get help from other states or hire a consultant; include any customizing details; require experienced project manager; include right to own test items • Link pattern of test development to general education cycle and include content experts in all aspects of test development (standards to test items) • Communicate with the field regularly through the process of developing extended content standards. Provide these standards to the field early. • If using any online pieces conduct pilot of online system to explore platform issues and get sign-off from someone in charge at district level • Exercise caution when updating online system during test window • Include an errata page on online system for notifying field of updates • Have a backup server if you have online components • Remain cheerful! Redesign of Alternate Assessments

17. Georgia • Melissa Fincher Georgia Department of Education Redesign of Alternate Assessments

18. Purpose and Rationale • Why did Georgia redesign our alternate? • In a word: COMPLIANCE • In a few words: OPPORTUNITY TO LEARN Redesign of Alternate Assessments

19. Goals The GAA is designed to ensure that students with significant cognitive disabilities are: Provided access to the state-mandated curriculum. Given the opportunity to demonstrate progress toward achievement of curriculum knowledge, concepts, and skills. Redesign of Alternate Assessments

20. Development The GAA is a portfolio of student work provided as evidence that a student is making progress toward grade-level academic standards. Evidence provided must show instructional activities and student work that is aligned to specific grade-level standards. Redesign of Alternate Assessments

21. Development Georgia elected to go with a portfolio format to allow flexibility for a very diverse group of students. The portfolio format allows the teacher, who knows the student best, to design and document instructional tasks that are meaningful and purposeful for the individual student. Redesign of Alternate Assessments

22. Development • Collaboration between Assessment, Special Education, and Curriculum • Significant investment in training of educators surrounding curriculum access • Ongoing documentation of decisions surrounding development and implementation • Involvement of Georgia’s Technical Advisory Committee • Augmented with an AA-AAS expert Redesign of Alternate Assessments

23. Challenges • Documenting technical quality • Traditional indices do not lend themselves easily to alternate assessments • Federal Peer Review • Complicated assessment program • Teacher buy-in Redesign of Alternate Assessments

24. Lessons Learned • Investing in teacher training pays off, but it needs to be continual as the assessment evolves. • Technical documentation of alternate assessments looks different, but it is worth pursuing. • Validity is an ongoing journey. • Students are doing things we never thought possible. Redesign of Alternate Assessments

25. Louisiana • Claudia Davis Louisiana Department of Education Redesign of Alternate Assessments

26. Purpose and Rationale • Purpose: Louisiana’s alternate assessment did not receive approval in the USDOE peer review process. Redesign of Alternate Assessments

27. Goals • Louisiana’s alternate assessment (LEAP Alternate Assessment or LAA) lacked: • Academic focus (although linked to state standards) • Alignment with grades or grade spans • USDOE advised a redesign of the LAA to be implemented in spring of 2008 • LDE began the redesign in July of 2007 Redesign of Alternate Assessments

28. Development • First, Louisiana developed Extended Standards (ES), i.e., extensions of state content standards. • What should students with significant cognitive disabilities know and be able to do? • How do we identify expectations regarding the breadth and depth of the standards, benchmarks, and grade-level expectations (GLEs)? • How should students demonstrate knowledge and skills based on the GLEs? • What is the appropriate range of tasks to be used to measure this knowledge and these skills? Redesign of Alternate Assessments

29. Development • LDE contracted with the testing vendor to develop Extended Standards for ELA, Mathematics, and Science. • Recommendations from an initial committee of state special educators regarding selected standards and a template for the ESs, including Complexity Levels, were used to guide the development. • Committees of Louisiana educators (general and special educators) reviewed the proposed ESs with Complexity Levels. • Content-area groups across four grade spans (grades 3-4, 5-6, 7-8, and 9-11) • Review was time intensive -- one intensive week Redesign of Alternate Assessments

30. Development • Complexity Levels of the Extended Standards: • Each ES has 3 levels of complexity (1 being least difficult) that provide access to general education concepts and skills. • They serve as guidelines for the development of assessment tasks at 3 levels of complexity. • Math Example: 3. Add and/or subtract to solve simple problems. 2. Identify simple addition and subtraction concepts within daily living problems. 1. Count to solve simple problems. Redesign of Alternate Assessments

31. 3 is most complex Extended Standard: ES-8 /5 refers to GLE 8 from grade 3 ES-8/ 5 refers to GLE from grade 4 O O GLE from grade 3 (top) and grade 4 (bottom)

32. Development • Extended standards and complexity levels were finalized in late September. • Extended Standards Handbook (draft) disseminated to school districts in October. • Assessment development began. Redesign of Alternate Assessments

33. ELA, Math, Science ELA, Math ELA, Math, Science ELA, Math ELA, Math Science 3 and 4 5 and 6 7 and 8 High school 9 10 11 Alternate AssessmentsGrade SpanContent Area Redesign of Alternate Assessments

34. Alternate Assessments • 25 performance tasks per content and grade span • Each task administered and scored by the teacher • Tasks scored on a 0-1 point or 0-2 point scale using a rubric Redesign of Alternate Assessments

35. Development • Alignment Study • Alignment of performance tasks to the Extended Standards • Panel of eight expert independent reviewers (4 state reviewers/4 national reviewers) • Based on Webb’s Depth-of-Knowledge alignment process for use in aligning ES to performance tasks (2007) • Level 1: Recall of Information • Level 2: Basic Reasoning • Level 3: Complex Reasoning Redesign of Alternate Assessments

36. Challenges • Time was the biggest challenge. • Short time frame: 8 months between first development activity (July, 07) and administration window for the assessment (February–March, 08) • Lack of adequate time for development activities • Lack of preparation time for teachers to provide instruction and to prepare themselves and their students for the assessment • Lack of time for adequate delivery of professional development to district personnel • Lack of time to field test the items (2008 scores will not be included in state accountability results for schools and districts) Redesign of Alternate Assessments

37. Challenges • Other challenges • Scheduling collaborative meetings with colleagues in Special Populations on a short timeframe • Student IEPs completed based on former alternate assessment • Necessary changes in Participation Criteria for AA • Decisions regarding • Test format • Accommodations for students • Manipulatives for tasks • Assistive technology Redesign of Alternate Assessments

38. Lessons Learned • Adequate preparation time for students and teachers is a MUST! • Adequate professional development is a MUST! • The assessment is not appropriate for all students with significant cognitive disabilities (SCD). • Teachers of students with SCD are not so familiar with standardized testing and need more training on administration procedures and security. • Opinions from the field of the new assessment ranged from “loved it” to “hated it” based primarily on their students’ abilities to access the test. • Assistive technology personnel need to be included at the beginning of the development phase. Redesign of Alternate Assessments

39. To End on a Positive Some Teacher Comments: • “I’m so pleased to have a curriculum (Extended Standards) to guide my instruction.” • “The new test is easier to administer.” • “ A good test for content.” • “IEP goals will have to be rewritten to address the new assessment.” • “Great improvement.” • “Great design!” • “Pleasantly positively surprised.” • “The test reflected what I teach.” Redesign of Alternate Assessments

40. National and StatePerspectives…. And Next Steps • Rachel F. Quenemoen, Senior Research Fellow, NCEO National Center on Educational Outcomes Redesign of Alternate Assessments

41. NCEO STATE SURVEY REPORTS • 2005 State Special Education Outcomes: Steps Forward in a Decade of Change • 2003 State Special Education Outcomes: Marching On • 2001 State Special Education Outcomes: A Report on State Activities at the Beginning of a New Decade • 1999 State Special Education Outcomes: A Report on State Activities at the End of the Century Thompson & Thurlow (1999, 2001, 2003) Thompson, Johnstone, Thurlow, & Altman (2005) Redesign of Alternate Assessments

42. Content Addressed by Alternate Assessments: Change Over Time *Category possibly included grade level standards prior to 2005 ** Category introduced in 2005

43. 2005 - Outcomes Measured by Rubrics on Alternate Assessments 0 5 10 15 20 25 30 35 40 45 50 Skill/Competence_ 25 (40) Level of Assistance 25 (32 independence) 23 (23) Degree of Progress Number/Variety of Settings 20 (21) Alignment with Academic Content Standards 18 15 (18) Ability to Generalize Appropriateness 13 (20) Staff Support 10 (20) 10 Social Relationships Self Determination 9 Participation in General Education Settings 7 (12) 7 Support Number of Regular States (Numbers in parentheses from 2001)

44. Alternate Assessment Approaches 2000-2005 (from 2005 Survey) **Of these 25 states, 13 use a standardized set of performance/events/tasks/skills. ***Of these 7 states, three require the submission of student work.

45. Flexibility and Standardization • Nominal categories are NOT often useful for characterizing the technical aspects of the assessment (see Gong & Marion, 2006). • The evaluation of technical adequacy interacts with the types of alternate assessments (i.e., choices/ degree of flexibility-standardization) being employed. • This does NOT mean that standardization is good and flexibility is bad—it all depends on purposes! Redesign of Alternate Assessments

46. Where Are We Now? • Content coverage – National Alternate Assessment Center work – University of Kentucky: Is it reading? Is it math? Is it science?; University of North Carolina: Links for Academic Learning; other methodologies for alignment. Peer Review suggests great variability, near and far linkages, but a steady trend is toward academic content. Key questions: is it measureable bits or big ideas and concepts or both? What is grade level for these students? • Scoring criteriaand procedures – What does student performance look like? Student vs. system? How do we measure “independence?” Who scores? Who checks? Trust but verify? Flexibility vs. standardization issue. Peer Review suggests great variability on this. Concerns related to support, prompts, communications issues, academics interaction. Redesign of Alternate Assessments

47. Where Are We Now?Part 2 • Performance/achievement descriptors and standard setting • Is there achievement on the content? • Is the content clearly referenced? • How have accessibility issues been factored in? • What does independence mean? How good is good enough? What should these students know and be able to do? How well? This needs careful monitoring over time, consequential validity studies. • Approach • Degree and logic of flexibility and standardization choices Nominal categories are not particularly useful descriptors. Unfortunately, “…the naked eye is drawn to test format” not educational soundness (Baker, 2007) Redesign of Alternate Assessments

48. More or Less Than Meets the Eye? BECAUSE of the number of uncertainties still in play, we need: • Transparency • Integrity • Consequential validity studies • Planned improvement over time Redesign of Alternate Assessments

49. Questions Redesign of Alternate Assessments