Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

1. Designing Math and ELA Tasks for AA-AAS Using ECD and UDL Renée Cameto and Geneva Haertel – SRI InternationalWendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of Education Carol Scholtz – Idaho State Department of Education Deborah Matthews – Kansas State Department of Education Shawnee Wakeman – Moderator Edward Roeber – Discussant National Conference on Student Assessment June 19, 2011 • This material is based on work supported by two Enhanced Assessment Grants from the U.S. Department of Education: Alternate Assessment Design—Mathematics and English Language Arts. The PADI online system was developed through a grant from the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Department of Education or the National Science Foundation.

2. Introduction • Based on work from two Enhanced Assessment Grants funded by the U.S. Department of Education • Each project involved a consortium of states and SRI International: • Alternate Assessment Design–Mathematics • Utah, Idaho, Florida • Alternate Assessment Design–English Language Arts • Idaho, Utah, Kansas

3. Agenda • ECD Framework • Co-Design Process • Example Design Patterns and Tasks • Task Tryouts • Teacher training and materials • Videos of task administration • Results • Panel Discussion with State Representatives • Q & A • Discussant

4. Need • Alternate Assessments need to be well designed • The design and development processes employed need to be systematic, rigorous, and reflective of industry standards • NCLB/ESEA require that students with disabilities, including SWSCD, be assessed on grade-level academic content • Assessment continues to be a lever of educational change (assessment drives instruction and school reform)

5. Innovative Approach • Evidence-centered design (ECD) is an innovative assessment design process • Initial work on ECD conducted at ETS by Mislevy, Steinberg, and Almond (2003) • Has been used for more than 15 years; in these EAG projects, ECD is being extended to the population of students with significant cognitive disabilities • Is a framework and set of processes • Meets criteria for developing well-designed assessments • Supports the design of items that are aligned to the focal constructs of interest

6. Innovative Approach • ECD can be applied to: • All subject areas • All grade levels • All types of assessments (large scale, summative, formative, technology-enabled, pencil/paper, advanced placement, workforce, etc.) • All types of alternate assessments (portfolio, performance task, checklist, etc.) • All item/task formats (multiple choice, constructed response, performance task, etc.) • Integrating UDL into the ECD framework promotes accessibility of items through consideration of student needs and abilities during initial design and throughout the design process

7. What is Evidence-Centered Design? • Critical Question: How do we judge what students know and what they can do? • Make explicit what knowledge or skills are the target • Identify non-target but required knowledge and skills • Generate the kinds of observations that provide evidence of a student having the target knowledge • Determine the kinds of stimuli and work products needed to gather the evidence • These attributes form the basis of the assessment argument - an argument based on evidence

8. Co-Design Process • ECD process makes use of co-design at all layers • Co-design team typically includes: • Special educators (experience with SWSCD) • Content experts • Assessment specialists • State Department of Education administrators

9. Alternate Assessment Design • A multistep process using ECD co-design Common Core Create Summary Task Templates Author Assessment Tasks Select Standards Create Design Patterns Pilot and Refine Tasks Domain Modeling Domain Analysis Assessment Implementation Assessment Delivery Conceptual Assessment Framework

10. Select Standards GOAL: In Domain Analysis, content relevant to the assessment is organized and selected • Identify standards to meet state AA-AAS needs, for example, by identifying commonalities among the consortium states’ extended standards and comparison standards (e.g., NCTM Standards and Expectations, Common Core State Standards) • Develop list of standards common to all participating states Common Core Select Standards

11. Create Design Patterns GOAL: In Domain Modeling, a narrative description of the assessment argument structure (Design Pattern) is produced to guide task development • Design Patterns for selected standards are developed • Design Patterns are reusable and improve efficiency of task development • Design Patterns can improve contentvalidity • Design Patterns create a design space for assessment developers Create Design Patterns

12. Elements of Design Patterns • Student Model • What knowledge, skills or abilities (KSAs) should be assessed?

13. Elements of Design Patterns • Student Model • What knowledge, skills or abilities (KSAs) should be assessed?

14. Elements of Design Patterns • Evidence Model • What behaviors or performances should reveal the knowledge, skills, and abilities?

15. Elements of Design Patterns • Evidence Model • What behaviors or performances should reveal the knowledge, skills, and abilities?

16. Elements of Design Patterns • Task Model • What tasks, situations, or stimuli should elicit those behaviors and performances?

17. Create Design Patterns • Task Model • What tasks, situations, or stimuli should elicit those behaviors and performances?

18. Create Design Patterns • Variable Features are the aspects of the task that are varied to support the Additional KSAs

19. Create Design Patterns • Variable Features are the aspects of the task that are varied to support the Additional KSAs Source: Center for Applied Special Technology (CAST), http://CAST.org

20. Author Assessment Tasks GOAL: In Assessment Implementation, tasks are authored • Information pre-populates from the Design Pattern to the Task Template • Templates are used to author a family of itemsaligned with a Focal KSA • Complexity is varied within the family of items • UDL is incorporated • Items individually administered Author Assessment Tasks

21. Author Assessment Tasks • Step 1. Pre-populate Task Template with information from the Design Pattern • Focal KSAs and Additional KSAs • Potential Observations and Work Product • Variable Features for Cognitive Background and UDL • Characteristic Features • Step 2. Select attributes for authoring items • Focal KSAs and Additional KSAs • Potential Observations and Work Products • Variable Features for Cognitive Background and UDL

22. Author Assessment Tasks • Step 3. Select Variable Features to be used in the item family • Indicate how Variable Features will be applied (e.g., limit numbers to those with two or fewer digits, include multiple representations of stimulus materials) • Consider ways to vary complexity • Consider ways to incorporate UDL

23. Author Assessment Tasks • Step 4. Create Item Directive

24. Mathematics Item 1 • Examiner shows student a picture of three pizzas and says, Here are three pictures of parts of a pizza. • Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza? ¾

25. Author Assessment Tasks • Step 4. Create Item Directive

26. Mathematics Item 2 • Examiner presents student with two photos, one with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas. • Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza. ½

27. Author Assessment Tasks • Step 4. Create Item Directive

28. Mathematics Item 3a • Examiner presents to student an illustration of two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.

29. Mathematics Item 3b • If student does not respond to item 3a or responds incorrectly, examiner removes the first illustration (of a whole pizza) and shows student only the second illustration. Examiner says, [Look at/touch] the photo of part of a pizza.

30. Author Assessment Tasks • Step 5. Document correct answer

31. Author Assessment Tasks • Step 6. Describe stimulus materials

32. Author Assessment Tasks • Step 7. Describe materials for the examiner

33. ELA Item 1 • Examiner says, This passage is from Roll of Thunder, Hear My Cry. Examiner reads passage to student: Cassie tells Mama that a mob is going to come after a boy. Papa runs outside with a gun. Soon, Mama sees that their cotton field is on fire. She thinks it started by lightning. Then Cassie goes back to her room and wonders if people can live in peace. Everyone goes out to the cotton field. Papa and Mama's neighbors are helping them put out the fire. The people in the mob are helping put out the fire too. Cassie thinks that her Papa started the fire.

34. ELA Item 1 • Examiner lays out a card with the following question and reads the question to the student: How did Papa starting the fire change the story? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read. • The fire made the mob more angry and they came after the boy • The mob helped put out the fire and didn't hurt the boy • Cassie got scared of the fire and ran away from home

35. ELA Item 2 • Examiner says, This passage is from Roll of Thunder, Hear My Cry. Examiner reads passage to student:A mob is going to come after a boy. His friends are warned that the mob is coming. One of the friends sets his own cotton field on fire to lead the mob away. All the neighbors and the mob come together to put out the fire. The boy is not hurt.

36. ELA Item 2 • Examiner lays out a card with the following question and reads the question to the student: What happened next after the friend set the cotton field on fire? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read. • All the neighbors and the mob put out the fire • The boy’s friends are warned that the mob is coming • A mob decides to come after the boy

37. ELA Item 3a • Examiner places a picture card in front of student and reads the first sentence from the passage: Cassie goes to the store. • Examiner places the second picture card in front of student and reads the second sentence from the passage: The storekeeper yells at Cassie. • Examiner places the third picture card in front of student and reads the third sentence from the passage: The storekeeper makes Cassie get out of the store.

38. ELA Item 3a • Examiner removes all pictures and then presents student with two pictures (Picture 1 and Picture 3) and says, Look at these pictures. Which picture shows what happened after the storekeeper yelled at Cassie? ELA Item 3b • If student cannot or does not respond, examiner removes Picture 1 leaving only Picture 3 in front of student and says, [Look at/touch] the picture that shows what happened after the storekeeper yelled at Cassie.

39. Design Patterns • Library of Design Patterns and Tasks • AAD-Mathematics and AAD-ELA projects, when completed, will have produced approximately 50 Design Patterns and Task families–200 items • Spans grades 3–8 and high school • Math Design Patterns organized around NCTM Standards and Expectations • ELA Design Patterns organized around Common Core State Standards • Supports design of multiple-choice, scenario-based, and portfolio assessment formats

40. Benefits of Using ECD with UDL What states gain from applying this approach in assessment design: • Content-valid assessment tasks designed to align to state or national standards • Increased efficiency through systematic task design and development-reusable design patterns and task templates • Built-in documentation of design decisions using PADI (NSF and SRI development funding) • Variation in DOK and complexity • Attention to UDL throughout design process

41. Pilot and Refine Tasks GOAL: In Assessment Delivery, the newly developed assessment tasks are pilot tested and refined based on empirical results • All newly developed assessment tasks must be empirically studied to establish their feasibility, reliability, and validity • Methodologies may include cognitive labs, teacher surveys, field tests of tasks, and student observations Pilot and Refine Tasks

42. Math Task Try-outs Research Questions: • Task and Item Viability • Can the 3 items within the exemplar task be administered as designed? • Is the task clear to the student and to the teacher? • Appropriateness for a Range of Student Performance Levels • Can any of the students do the most difficult item? • Can most students do the least difficult item?

43. Math Task Try-outs • Detailed Task Materials and Instructions • Data Collection Booklet: 85 questions • Score for each item administered • Item related information • Student received instruction related to the item • Item characteristics – directions, graphics, manipulatives and materials, language, complexity • Student information • Communication level, disability, grade-level • Teacher information • Years of experience, familiarity with academic standards

44. AAD-M Idaho - One State’s Experience with PADI & ECD PADI and ECD Pluses! • Thorough Stepwise Methodology –for AA ECO Based Item Development; based upon DOK, UDL fits with LAL • Stakeholder Participation – Assessment, Curriculum & Ed • SRI – Pioneering Methodology – AA Crosswalk with Multiple State ECOs (precursor to Common Core) • Increased # Tools in Educators’ Toolboxfor formative classroom assessment • Increased Educator Awareness of AA and Ties to Instruction

45. Flow Chart How to Administer Task Items • There are two paths: • Student responds correctly to A1 • Student responds incorrectly to A1

46. Sequence of Items within a Task: General Case When Item A1 is Correct

47. Sequence of Items within a Task: Special Case When Item A1 is Incorrect

48. Idaho AAD-M Task Try-out Incentives! • Stipend for Teachers Assessment and Instructional Materials • Student Rewards Shopping & Budgeting with Double Digit Subtraction

49. Math Task Try-outs • Number of student participants: 192 • Number of teacher participants: 55 • Number of tasks administered by state: 16 • 9 tasks/suites of items common across all states • 7 tasks/suites of items unique to each state

50. Math Task Try-outs • Fidelity of Implementation • Of the 1,547 item administrations: • 98% (1,514) items were administered as intended • 2% (29) items were not administered as intended