research on universal design of assessments l.
Download
Skip this Video
Download Presentation
Research on Universal Design of Assessments

Loading in 2 Seconds...

play fullscreen
1 / 51

Research on Universal Design of Assessments - PowerPoint PPT Presentation


  • 120 Views
  • Uploaded on

Research on Universal Design of Assessments. Making Assessments Accessible and Valid for All Students. Universal Design of Assessment: The Whole Elephant. David Malouf Office of Special Education Programs U.S. Department of Education. Universal Design of Assessments.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Research on Universal Design of Assessments' - chandelle


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
research on universal design of assessments
Research on UniversalDesign of Assessments

Making Assessments Accessible

and Valid for All Students

National Center on Educational Outcomes

universal design of assessment the whole elephant

Universal Design of Assessment:The Whole Elephant

David Malouf

Office of Special Education Programs

U.S. Department of Education

National Center on Educational Outcomes

universal design of assessments
Universal Design of Assessments

Designing assessments to be inherently accessible and valid for the widest possible range of students.

National Center on Educational Outcomes

series of federal laws
Series of Federal Laws
  • Improving America’s Schools Act of 1994
  • 1997 reauthorization of the Individuals with Disabilities Education Act
  • No Child Left Behind Act of 2001
  • Upcoming reauthorization of the Individuals with Disabilities Education Act

National Center on Educational Outcomes

improving america s schools act of 1994
Improving America’s Schools Act of 1994
  • Under Title I, all students were to be included in assessments, public reporting, and accountability.

National Center on Educational Outcomes

individuals with disabilities education act 1997 reauth
Individuals with DisabilitiesEducation Act (1997 reauth.)
  • Added requirements for students with disabilities to be included in State and district-wide assessments.
  • Introduced alternate assessments
  • Required public reporting of aggregated and disaggregated data

National Center on Educational Outcomes

no child left behind act of 2001
No Child Left Behind Act of 2001
  • Increased assessment demands
  • Increased requirements for including students with disabilities and students with limited English proficiency
  • Accountability for subgroups

National Center on Educational Outcomes

from the regulations for the no child left behind act
From the Regulations for theNo Child Left Behind Act:

§200.2 State responsibilities for assessment

(b) The assessment system required under this section must meet the following requirements:

.......

(2) Be designed to be valid and accessible for use by the widest possible range of students, including students with disabilities and students with limited English proficiency.

National Center on Educational Outcomes

president s commission on excellence in special education
President’s Commission on Excellence in Special Education

Created in 2001 to recommend reforms in America’s special education system

13 Public hearings and meetings, plus written input

Reported in July 2002

National Center on Educational Outcomes

president s commission on excellence in special education10
President’s Commission on Excellence in Special Education

RECOMMENDATION—INCORPORATE UNIVERSAL DESIGN IN ACCOUNTABILITY TOOLS: Ensure all tools used to assess students for accountability and the assessment of progress are designed to include any accommodations and modifications for students with disabilities.

National Center on Educational Outcomes

upcoming reauthorization of the individuals with disabilities education act idea
Upcoming reauthorization of the Individuals with Disabilities Education Act (IDEA)

?

National Center on Educational Outcomes

senate bill 1248
Senate Bill 1248

Universal design. The State educational agency (or, in the case of a districtwide assessment, the local educational agency) shall, to the extent possible, use universal design principles in developing and administering any assessments under this paragraph.

§612(a)(16)(A)(ii)(E)

National Center on Educational Outcomes

the pressure is on
The pressure is on...

Will we do quick retrofits of tests and called them “universally designed”?

Will we simply port tests onto computers and call them “universally designed”?

National Center on Educational Outcomes

slide14

Constructs designed for accessibility

Inclusive Population

Accessible items

Amenable to accommodations

Simple and clear instructions and procedures

Maximum readability

Maximum legibility

National Center on Educational Outcomes

three research projects on universal design of assessment
Three Research Projects on Universal Design of Assessment

“Development Techniques for Universally Designed Assessments”

Sandra Thompson, University of Minnesota

“Access to Assessment via Technology”

Jacqueline Kearns, University of Kentucky

Bob Dolan, CAST

“Project MAP (Making Accommodations Personalized”

Gerald Tindal, University of Oregon

National Center on Educational Outcomes

slide16

Development Techniques for Universally Designed Assessments

National Center on Educational Outcomes

University of Minnesota

Sandy Thompson

http://education.umn.edu/nceo

National Center on Educational Outcomes

3 studies
3 Studies
  • Use “think aloud” to examine student – end user - perspectives
  • Analyze items for differences possibly due to design features
  • Design training for item/test reviewers

National Center on Educational Outcomes

think aloud
“Think aloud”
  • Recently interviewed 90 students using think aloud protocol
  • 4th and 8th grade
  • Used multiple choice and constructed response items from state math test

National Center on Educational Outcomes

logistics
Logistics
  • 10 researchers
  • 5 days
  • Worked in pairs
  • All sessions videotaped
  • Primary accommodations included oral administration and sign language interpretation

National Center on Educational Outcomes

student characteristics
Student Characteristics

National Center on Educational Outcomes

overall observations
Overall Observations
  • Students who were confident of content did not have problems with design
  • Students who had no idea how to solve the problem did not have problems with design
  • Students “in the middle” – not sure of content, some reading difficulty, design made a difference

National Center on Educational Outcomes

examples of student perceptions
Examples of Student Perceptions
  • Many students didn’t see one of the cities on a map
  • The name of one of the cities was “Independence” - uncommon meaning
  • Box between top and bottom of item – some students did not read entire item
  • Sign for parallel gave away the answer
  • Some students read fraction 3 5/8 as “35 divided by 8”
  • Students unfamiliar with settings – “Glee club does number,” “fitness club”

National Center on Educational Outcomes

other observations
Other Observations
  • Some students got a lot of “help” from sign language interpreters and teachers who wanted to make sure they understood the problem – what happens on test day?
  • Some student forms reported need for oral administration when they could clearly read the items independently – are some accommodations inappropriate?
  • For English language learners - is oral administration in English an appropriate accommodation on a Math test?
  • Should sign language interpreters have a script to follow so they don’t “give away” some answers?

National Center on Educational Outcomes

considerations for item review
Considerations for Item Review
  • Overall appearance is clean and organized
  • Clear format for text
  • Clear format for pictures and graphics (when essential to item)
  • Concise and readable text
  • Format allows for changes without changing meaning or difficulty
  • Meets criteria for measuring what it is intended to measure

National Center on Educational Outcomes

considerations for test review
Considerations for Test Review
  • Meets general criteria for measuring what it is intended to measure
  • Overall appearance is clean and organized
  • Instructions are necessary, clear, and understandable
  • Scoring criteria are appropriate
  • Others?

National Center on Educational Outcomes

alignment and usability need to be considered together
Alignment and Usability Need to be Considered Together

National Center on Educational Outcomes

future plans
Future Plans
  • Interview students with vision impairments
    • Partner with American Printing House for the Blind
  • Explore differential item analyses
    • Partner with researchers from CTB/McGraw-Hill
  • Produce short awareness video
    • Partner with Institute on Community Integration at University of Minnesota
  • Design and pilot training for item reviewers and item developers
    • Partner with Missouri Department of Education

National Center on Educational Outcomes

universal design of assessment applications of technology

Universal Design of Assessment: Applications of Technology

CCSSO National Conference onLarge-Scale Assessment

June 23, 2003

National Center on Educational Outcomes

universal design for learning 1
Universal Design for Learning (1)
  • Universal Design for Learning (UDL) extends the concept of universal design from a physical space to a cognitive space
  • Based upon psychological and neuroscientific theories of learning
  • Relies on scaffolds, supports, & accommodations which support students’ challenges and thus provide access to learning
  • Applies to design and development of goals, methods, assessments, and materials

National Center on Educational Outcomes

universal design for learning 2
Universal Design for Learning (2)
  • To support individual differences, students must be provided with multiple means of interacting with curriculum
    • Multiple means of representation
    • Multiple means of interaction and expression
    • Multiple means of engagement
  • Digital technology is neither necessary nor sufficient for UDL, but it is an enabling factor

National Center on Educational Outcomes

kentucky
Kentucky
  • Universal Design for Learning Expert Group
  • Instructional Technology for Student Success (ITSS) initiative
  • Digital Text Network
  • E-Text Schools program
  • Senate Bill 243
    • Requires any textbook offered for adoption in Kentucky schools be made available by the publisher in an accessible digital format

National Center on Educational Outcomes

ky cats online assessment
KY CATS Online Assessment
  • Web-based, individualized assessment for qualifying students with disabilities:
    • Students with IEP or 504 Plan that specifies need for "reader" as an instructional and assessment accommodation;
    • Students who require and routinely use text-reader or screen-reader technologies to access printed material in classroom instruction and assessment;
    • Students who have accessed and used the CATS Online Practice Area.
  • Based upon success of pilot studies, 16 districts, 31 Schools, & 204 students participated in “live” CATS Online this spring

National Center on Educational Outcomes

cats online
CATS Online

National Center on Educational Outcomes

maine
Maine
  • Maine Learning Results emphasize inclusion of diverse students by separating goals and methods
  • Maine Comprehensive Assessment System uses variety of state and local components, allowing flexibility and multiple measures of learning to accommodate learner diversity
  • State assessment (MEA) tightly aligned with Learning Results
  • Actively pursuing UDL as a means for increasing access to general curriculum

National Center on Educational Outcomes

overarching research question
Overarching Research Question

How can digital technologies be used to improve accessibility of large-scale assessments in accordance with the principles of Universal Design for Learning?

National Center on Educational Outcomes

research question 1
Research Question 1

What are the practical, procedural, and political design and implementation features of accessible computer-based assessments?

National Center on Educational Outcomes

research question 2
Research Question 2

What technology pre-requisite skills do students need to use computer-based assessments?

National Center on Educational Outcomes

research question 3
Research Question 3

What is the impact of a computer-based universally designed assessment on the scores of students with disabilities?

National Center on Educational Outcomes

research question 4
Research Question 4

To what extent does accessible curriculum design impact student assessment results?

National Center on Educational Outcomes

research question 5
Research Question 5

Can the universally designed assessment design features be replicated with new content in a different state context?

National Center on Educational Outcomes

research methodology
Research Methodology
  • Qualitative Studies
    • Interviews, observations, focus groups
  • Quantitative Studies
    • Survey, score analysis
  • Critical Theory
    • Constituent reflection

National Center on Educational Outcomes

preliminary findings to date
Preliminary Findings To Date
  • Interviewed approximately 40 students from 7 schools who took CATS Online
  • Most students used the ‘text-to-speech’ feature
  • Students liked the independence that the text reader provided, which allowed them to re-read questions multiple times

National Center on Educational Outcomes

features that promote accessibility
Features that Promote Accessibility
  • Multiple means of representation
    • Text-to-speech
    • Font features (e.g. size)
    • Screen layout options
    • Textual description of images
  • Multiple means of expression and interaction
    • Word processing
    • Flexible navigation

National Center on Educational Outcomes

features that impede accessibility
Features that Impede Accessibility
  • Word processing – requires a set of skills that some students may not have
  • Text-to-speech – voice quality and accuracy issues
  • Layout issues – poetry, tables, etc.

National Center on Educational Outcomes

preliminary item analysis positive exemplars
Preliminary Item AnalysisPositive Exemplars

National Center on Educational Outcomes

slide46

MEA Grade 11

Science and Technology

16. This diagram shows that the beams of light from two flashlights can pass through each other and then continue on unaffected. This observation illustrates which property or properties of light?

A. only particle

 B. only wave

C. both particle and wave

D. neither particle nor wave

Consider

  • Explicit reference to image
  • Redundancy of information between image and text

Learning Results: H-1, Energy. Students will understand concepts of energy. Students will be able to analyze the evidence that leads scientists to conclude that light behaves somewhat like a wave and somewhat like a particle.

National Center on Educational Outcomes

slide47

Scales of Justice

Use the illustration below to answer question 12.

CATS Grade 8 Social Studies

Consider

  • Explicit reference to image
  • Redundancy of information between image and text
  • Multiple opportunities for response (writing vs. keyboarding)

12. “Lady Justice” is a symbol of the United States justice system. As shown in the illustration above, she wears a blindfold and carries a balance scale. Explain how this symbol represents two characteristics of the U.S. justice system.

National Center on Educational Outcomes

preliminary item analysis negative exemplars
Preliminary Item AnalysisNegative Exemplars

National Center on Educational Outcomes

slide49

MEA Grade 11 Math

2. Which of the following is the best estimate of the sum of

.√26 + √78?

A. 10

 B. 14

C. 52

D. 104

Consider

  • Ability of text-to-speech tools to recognize and represent the square-root symbol
  • Potential invalidation of item when square-root symbol is “decoded” by human or digital reader

Learning Results: B-1, Computation. Students will understand and demonstrate computation skills. Students will be able to use various techniques to approximate solutions, determine the reasonableness of answers, and justify the results.

National Center on Educational Outcomes

slide50

Alex’s Garden

CATS Grade 8 Math

Consider

  • Transferring of stimulus to paper for response
  • Use of text-to-speech with image labels
  • Representation of image to blind students

30. Alex is watering part of his garden with a sprinkler that covers a circular area shown in the diagram below.

a. What is the total area of Alex’s garden? Show your work.

b. What is the area of the part of the garden that is being watered by the sprinkler? Express your answer to the nearest square foot. Show your work.

c. What percent of his garden is being watered by the sprinkler? Express your answer to the nearest percent. Show your work.

BE SURE TO LABEL YOUR RESPONSES (a), (b), and (c).

National Center on Educational Outcomes

concerns conclusions
Concerns & Conclusions
  • Applying universal design to instruction and assessment: which comes first? (Chicken and egg problem)
  • Practical considerations of use of technology e.g. security, scalability, cost
  • Need for seamless integration of instructional and assessment technologies to ensure valid, accurate, and fair testing

National Center on Educational Outcomes

ad