1 / 78

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL. Renée Cameto and Geneva Haertel – SRI International Patricia Almond – University of Oregon Wendy Carver – Utah State Office of Education Karen Denbroeder – Florida Department of Education

coby
Download Presentation

Designing Math and ELA Tasks for AA-AAS Using ECD and UDL

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Designing Math and ELA Tasks for AA-AAS Using ECD and UDL Renée Cameto and Geneva Haertel – SRI International Patricia Almond – University of Oregon Wendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of Education Carol Scholtz – Idaho State Department of Education Deborah Matthews – Kansas State Department of Education Bob Dolan– Discussant National Conference on Student Assessment June 27, 2012 • This material is based on work supported by two Enhanced Assessment Grants from the U.S. Department of Education: Alternate Assessment Design—Mathematics and English Language Arts. The PADI online system was developed through a grant from the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Department of Education or the National Science Foundation.

  2. Introduction • Based on work from two Enhanced Assessment Grants funded by the U.S. Department of Education • Each project involved a consortium of states and SRI International: • Alternate Assessment Design–Mathematics • Utah, Idaho, Florida • Alternate Assessment Design–English Language Arts • Idaho, Utah, Kansas

  3. Agenda • ECD/UDL Frameworks • Co-Design Process • Example Design Patterns and Item Suites • Math • English language arts (ELA) • Item Suite Tryouts • Teacher and student background characteristics • Student performance • Teacher feedback on items • Discussant • Q & A

  4. Need • Alternate Assessments need to be well designed • The design and development processes employed need to be systematic, rigorous, and reflective of industry standards • NCLB/ESEA require that students with disabilities, including SWSCD, be assessed on grade-level academic content • Assessment continues to be a lever of educational change (assessment drives instruction and school reform)

  5. Innovative Approach • Evidence-centered design (ECD) is an innovative assessment design process • Initial work on ECD conducted at ETS by Mislevy, Steinberg, and Almond (2003) • Has been used for more than 15 years; in these EAG projects, ECD is being extended to the population of students with significant cognitive disabilities • Is a framework and set of processes • Meets criteria for developing well-designed assessments • Supports the design of items that are aligned to the focal constructs of interest

  6. Innovative Approach • ECD can be applied to: • All subject areas • All grade levels • All types of assessments (large scale, summative, formative, technology-enabled, pencil/paper, advanced placement, workforce, etc.) • All types of alternate assessments (portfolio, performance task, checklist, etc.) • All item/task formats (multiple choice, constructed response, performance task, etc.) • Integrating UDL into the ECD framework promotes accessibility of items through consideration of student needs and abilities during initial design and throughout the design process

  7. What is Evidence-Centered Design? • Critical Question: How do we judge what students know and what they can do? • Make explicit what knowledge or skills are the target • Identify non-target but required knowledge and skills • Generate the kinds of observations that provide evidence of a student having the target knowledge • Determine the kinds of stimuli and work products needed to gather the evidence • These attributes form the basis of the assessment argument - an argument based on evidence

  8. Co-Design Process • ECD process makes use of co-design at all layers • Co-design team typically includes: • Special educators (experience instructing SWSCD in academics) • Content experts • ECD experts • State assessment administrators

  9. Alternate Assessment Design • A multistep process using ECD co-design Common Core Create Summary Task Templates Author Assessment Tasks Select Standards Create Design Patterns Pilot and Refine Tasks Domain Modeling Domain Analysis Assessment Implementation Assessment Delivery Conceptual Assessment Framework

  10. Select Standards GOAL: In Domain Analysis, content relevant to the assessment is organized and selected • Identify standards to meet AA-AAS needs • AAD-Math: Used NCTM Standards and Expectations later created crosswalk to Common Core State Standards • AAD-ELA: Common Core State Standards and the North Carolina Common Core State Standards Common Core Select Standards

  11. Create Design Patterns GOAL: In Domain Modeling, a narrative description of the assessment argument structure (Design Pattern) is produced to guide task development • Design Patterns for selected standards are developed • Design Patterns are reusable and improve efficiency of task development • Design Patterns can improve contentvalidity • Design Patterns create a design space for assessment developers Create Design Patterns

  12. PADI Online Design System

  13. Elements of Design Patterns • Student Model • What knowledge, skills or abilities (KSAs) should be assessed?

  14. Elements of Design Patterns • Student Model • What knowledge, skills or abilities (KSAs) should be assessed?

  15. Elements of Design Patterns • Evidence Model • What behaviors or performances should reveal the knowledge, skills, and abilities?

  16. Elements of Design Patterns • Evidence Model • What behaviors or performances should reveal the knowledge, skills, and abilities?

  17. Elements of Design Patterns • Task Model • What tasks, situations, or stimuli should elicit those behaviors and performances?

  18. Elements of Design Patterns • Task Model • What tasks, situations, or stimuli should elicit those behaviors and performances?

  19. Relationship of Design Pattern Elements • Variable Features are the aspects of the task that are varied to support the Additional KSAs

  20. Relationship of Design Pattern Elements • Variable Features are the aspects of the task that are varied to support the Additional KSAs Source: Center for Applied Special Technology (CAST), http://CAST.org

  21. Author Assessment Tasks GOAL: In Assessment Implementation, tasks are authored • Information pre-populates from the Design Pattern to the Task Template • Templates are used to author a family of 4 itemsaligned with Focal KSA • Complexity is graduated within the family of four items • UDL is incorporated • Items individually administered Author Assessment Tasks

  22. Author Assessment Tasks • Step 1. Pre-populate Task Template with information from the Design Pattern • Focal KSAs • Additional KSAs • Potential Observations and Work Product • Variable Features for Cognitive Background and UDL • Characteristic Features • Step 2. Select attributes for authoring items • Focal KSA • Additional KSAs • Potential Observations and Work Products for each item • Variable Features for Cognitive Background and UDL

  23. Author Assessment Tasks • Step 3. Select Variable Features to be used in the item family • Consider ways to vary complexity (e.g., limit numbers to those with two or fewer digits, limit number of words in a passage • Consider ways to incorporate UDL (e.g., include multiple representations of stimulus materials)

  24. Author Assessment Tasks • Step 4. Create Item

  25. Mathematics Grade 9-12 Recycling

  26. Mathematics Item C – part 1

  27. Mathematics Item C – part 2

  28. Mathematics Item C – part 3

  29. Mathematics Item B

  30. Mathematics Item A1

  31. Mathematics Item A1

  32. Author Assessment Tasks • Step 5. Document correct answer • Step 6. Describe stimulus materials • Step 7. Describe materials for the examiner

  33. ELA Reading Grade 3 Charlotte’s Web

  34. ELA Item C- part 1 Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said, "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend. tired sad excited

  35. ELA Item C- part 2 Wilbur, the pig, lived in a big barn. He lived with horses, geese, sheep, and a rat. Even so, Wilbur was lonely. He started to cry. Then all of a sudden, a voice said, "I will be your friend." She said her name was Charlotte. Wilbur liked having a new friend. happy sad scared

  36. ELA Item B Wilbur watched Charlotte, the spider, spin her web. Wilbur wished he could make a web like Charlotte. He could not because he was a pig, not a spider. Spin a web Eat food Crawl like a spider

  37. ELA Item A1 Wilbur, the pig, lived in a big barn

  38. ELA Item A2 Wilbur, the pig, lived in a big barn

  39. Library of Design Patterns and Items Suites • AAD-Math and AAD-ELA projects produced: • 50 Design Patterns • 50 Item Suites – 200 items • Spans grades 3–8 and high school • Math Design Patterns organized around NCTM Standards and Expectations cross-walked to CCSS • ELA Design Patterns organized around Common Core State Standards • Using multiple-choice, open-ended and scenario-based formats

  40. Benefits of Using ECD with UDL What is gain from applying this approach in assessment design: • Content-valid assessment tasks designed to align to state or national standards • Increased efficiency through systematic task design and development-reusable design patterns and task templates • Built-in documentation of design decisions using PADI (NSF and SRI development funding) • Variation in complexity and DOK • Attention to UDL throughout design process

  41. Pilot and Refine Tasks GOAL: In Assessment Delivery, the newly developed assessment tasks must be pilot tested and refined based on empirical results • All newly developed assessment tasks must be empirically studied to establish their feasibility, reliability, and validity • Methodologies may include cognitive labs, teacher surveys, field tests of tasks, and student observations Pilot and Refine Tasks

  42. Item Suite Tryouts Research Questions • Task and Item Viability • Can the suite of items be administered as designed? • Are the items clear to the student and to the teacher? • Appropriateness for a Range of Student Performance Levels • Can any of the students do the most difficult item? • Can most students do the least difficult item?

  43. Item Suite Tryouts • Detailed Materials and Instructions for teachers • Teacher completes Data Collection Booklet: • Performance for each student on each item administered • Item related information • Student received instruction related to the item • Item viability–appropriateness of directions, graphics, manipulatives and materials, language, complexity • Student background information • Communication level, disability, grade-level • Teacher background information • Years of experience, familiarity with academic standards

  44. Flow Chart Start here How to Administer Suite of Items • There are two paths: • Student responds correctly to A1 • Student responds incorrectly to A1

  45. Sequence of Items within a SuiteWhen Item A1 Is Answered Correctly

  46. Sequence of Items within a SuiteWhen Item A1 Is Answered Incorrectly

  47. School Types and Teachers Degrees

  48. Percent of Teachers with 5 or More Years Experience

  49. Item Suite Tryout Study Sample

  50. Percent of Students at Three Levels of Communication

More Related