slide1
Download
Skip this Video
Download Presentation
Assessment Architecture: Building Universally Designed Large-Scale Assessments

Loading in 2 Seconds...

play fullscreen
1 / 62

National Center on Educational Outcomes - PowerPoint PPT Presentation


  • 110 Views
  • Uploaded on

Assessment Architecture: Building Universally Designed Large-Scale Assessments. CCSSO Preconference Clinic Saturday, June 21 1:00 – 5:00pm. Goals for Today. Identify and give examples of key elements of universally designed assessments

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'National Center on Educational Outcomes' - lexine


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Assessment Architecture: Building Universally Designed Large-Scale Assessments

CCSSO

Preconference Clinic

Saturday, June 21

1:00 – 5:00pm

National Center on Educational Outcomes

goals for today
Goals for Today
  • Identify and give examples of key elements of universally designed assessments
  • Use assessment results to determine whether items are universally designed
  • Apply considerations for item review to sample test items

National Center on Educational Outcomes

goals for today3
Goals for Today
  • Explore universally designed assessments in state assessment RFPs and Test Specifications
  • Know where to go for information, resources, and support

National Center on Educational Outcomes

agenda
Agenda
  • Building Design: Form Follows Function (and Taste!)
  • Welcome from NCEO!
  • Foundations of Universally Designed Assessments
  • Break
  • Measure Twice, Cut Once
  • Check Out the Materials
  • Keep it on the Level
  • Nail Down the Bids
  • Final Inspection

National Center on Educational Outcomes

slide5

Title I Regulationsintroduce the need for universally designed assessments –

[Assessments must be] designed to be accessible and valid with respect to the widest possible range of students, including students with disabilities and students with limited English proficiency.

Sec. 200.2(b)(2)

National Center on Educational Outcomes

slide6

Caution

While universally designed assessments can make tests more equitable, producing results that are more valid for all students, they cannot replace instructional opportunity!

National Center on Educational Outcomes

slide7

Elements of Universally Designed Assessments

  • Inclusive assessment population
  • Precisely defined constructs
  • Items developed and reviewed for bias and accessibility
  • Amenable to accommodations

National Center on Educational Outcomes

slide8

Elements of Universally Designed Assessments

  • Simple, clear, and intuitive instructions and procedures
  • Maximum readability/ comprehensibility
  • Maximum legibility: text, graphs, tables, illustrations, and response formats

National Center on Educational Outcomes

slide9

Inclusive Assessment Population

  • Universally designed assessments:
  • Consider all types of students in the general curriculum from the beginning
  • Include students with disabilities and ELLs in item tryouts and field testing

National Center on Educational Outcomes

slide10

Universally designed assessments reflect good measurement qualities:

  • Actually measure what they are intended to measure
  • Remove all non-construct-oriented cognitive, sensory, emotional, and physical barriers

National Center on Educational Outcomes

slide11

Is the use of “hold” as a noun familiar to students?

Is the concept of a “rock climbing wall” familiar to most students?

Will students be distracted by the odd shapes on the diagram?

“Four holds on one of the rock climbing walls are labeled on the diagram below. Matthew first climbs vertically 10 feet from Hold A to Hold B, horizontally 25 feet from Hold to Hold C, and then vertically 15 feet from Hold C to Hold D. How many fewer feet would Matthew have climbed if he had climbed directly from Hold A to Hold D?”

National Center on Educational Outcomes

slide12

Amenable to Accommodations

  • Universally designed assessments allow needed accommodations to be used
  • Plan for students who continue to need accommodations
  • Facilitate the use of accommodations such as Braille, assistive technology, bilingual dictionaries or translations

National Center on Educational Outcomes

american printing house for the blind

American Printing House for the Blind

Accessible Test Department

National Center on Educational Outcomes

aph s commitment
APH’s Commitment
  • Provide high quality tests in accessible formats for students with visual impairments
  • Build understanding of accessibility in testing students with visual impairments

National Center on Educational Outcomes

braille issues
Braille Issues
  • Pictures
  • Graphics
  • Appropriate test items.

National Center on Educational Outcomes

print issues
Print Issues
  • Photocopying
  • Use of gray scale
  • Measurement items

National Center on Educational Outcomes

we promote
We Promote…
  • Using VI expert during test item development
  • On time tests and practice materials
  • Teaching skills that students need

National Center on Educational Outcomes

we have plans
We have plans…
  • Publish manual on making tests accessible for VI
  • Research on what works!
  • Test publisher workshop
  • State assessment personnel workshop

National Center on Educational Outcomes

we can do this
We Can Do This
  • Have VI students taking and passing high standards tests
  • Have access to tests in formats needed, on time and of high quality
  • Raise the expectations of the general public

National Center on Educational Outcomes

slide20

Before you criticize someone, walk a mile in his shoes….

…then when you do criticize that person, you’ll be a mile away and have his shoes!!

National Center on Educational Outcomes

assessments designed to better include english language learners benefit all types of kids
Assessments designed to better include English language learners benefit all types of kids!
  • Students have the experience to understand the items
  • Language is clear, simple and indicates precisely what is required from student (“Plain language”)
  • Questions are amenable to supports that ELLs might use
  • Cognitive demands are reasonable

National Center on Educational Outcomes

slide22
“While writers might think certain expectations are obvious, if they are not explicit in the item, then they are subject to honest misinterpretation in the responses.”

(Kopriva, 2000, p. 39)

National Center on Educational Outcomes

slide23

Experience can be crucial!

To raise money for a trip to the Wolfridge Environmental Learning Center, sixth graders at Johnson Middle School are selling raffle tickets. The raffle prize is an electric scooter worth more than $300. A total of 500 tickets were sold. You bought two raffle tickets, your sister bought three and your father bought one. What is the probability that someone in your family will win the prize?

National Center on Educational Outcomes

recommendations to improve accessibility of text kopriva 2000
Recommendations to Improve Accessibility of Text (Kopriva, 2000)
  • Simple, brief and consistent sentence structure in items
  • Consistent and clear paragraph structure
  • Present tense and active voice
  • Minimal paraphrasing and rewording. If used, identify the original statement in parentheses
  • Minimal use of pronouns. Follow a pronoun with the term it refers to in parentheses
  • High frequency words
  • Avoid words with double meanings and colloquialisms. If used, define them in the text.

National Center on Educational Outcomes

slide25
Young historians take projects to the granddaddy of museumby Jennifer Corbett, Staff CorrespondentStar Tribune

Para. 1 “When Nicole Zachor, Laura Swanson and Carol Hinz started work on a project for history class a few months ago, the White Bear Lake sophomores had no idea that it would be displayed at the Smithsonian Institution’s National Museum of American History…”

Para. 12 “ This year junior and senior high students started work on their projects in January or February. A project can be a research paper but it can also be a group of individual media presentation, display presentation or performance”

National Center on Educational Outcomes

slide26
Which of the following is a condition for a student to participate in National History Day?

The student must be a junior or senior in high school.

The student must be able to go to Washington, D.C.

C. The student must do a project related to the national topic.

D. The student must do the project on his or her own by himself or herself.

¿Cuál de las siguientes es una condicíon para que un estudiante participe en el Día Nacional de Historia?

Los estudiantes deben estar en grados once o doce de High School.

El estudiante debe estar en posibilidad de ir a Washington, D.C

El estudiante debe hacer un proyecto relacionado a un tema nacional.

El estudiante debe hacer el proyecto por sí solo.

Think about accommodations up front!

National Center on Educational Outcomes

cognitive demands
Cognitive Demands
  • Amount of text not relevant to items
  • Length of text
  • Number of long texts
  • Timing (may be unspoken)
  • Amount of unfamiliar words
  • Placement of definitions (in text, to side, separate)
  • Location of native language text
  • Computerized/Hypertext

National Center on Educational Outcomes

preliminary research in universal design
Preliminary Research in Universal Design
  • Sample of 230 students taken from four schools in U.S. Southwest.
  • Two schools were “town” schools (pop. 20,000) and two were “rural” schools.
  • Students chosen from sixth grade teams that had populations of students with disabilities.

National Center on Educational Outcomes

slide29

Research Design

  • Two tests were created, one from sample statewide test items, the other re-designed using UD principles.
  • Each student took both tests.
  • Students randomly assigned to take a particular test first to prevent practice effect.
  • Constructs held constant for each item.

National Center on Educational Outcomes

slide30

Community Involvement

  • Advisory Board trained in principles of Universal Design and asked to comment / suggest improvements based on their perspectives.
  • Team consisted of three parents of children in special education program (one Navajo, one Latina, one Anglo) and one community member with dyslexia.

National Center on Educational Outcomes

slide31

Sample Original Item

Ramón is building a doghouse. He wants the roof of the doghouse to be at an angle that is more than 90° but less than 110°. Which angle below could he use for the roof?

A. B.

C. D.

National Center on Educational Outcomes

revised item
Revised Item

Which angle is more than 90° and less than 110°?

A. B.

C. D.

National Center on Educational Outcomes

what changed
What changed?

Design element #2: Construct more precisely defined.

Design element #3: Bias eliminated (dog house, Ramón)

Design element #4: “Built in accommodations” – un-timed, students circled answer on paper, did not bubble

Design element #5: Simple instructions and procedures

Design element #6: More comprehensible language

Design element #7: Larger font

National Center on Educational Outcomes

slide34

Results

  • Means of two tests were compared and t-tests performed.
  • A difference of 8.16 (1.67 sig.) was found between means, a statistically significant finding.
  • Effect size calculated using Cohen’s d. Effect of design = .061 (or 6/10 Standard Deviation difference) – a “moderate effect”

National Center on Educational Outcomes

slide35

Why??

  • Students with largest difference between two tests were interviewed to determine difference for them.
  • Students noted that: more direct language made it easier for them to “understand” items and unlimited time helped them to “think better” about items. Students also said they “remembered” content better on UD test.

National Center on Educational Outcomes

what have we learned
What have we learned?
  • Design matters!! How a test is designed may affect how a student scores on that test.
  • Items that are better designed appear to aid students who are English Language Learners with disabilities “show what they know” better.
  • This leads to more valid assessment of traditionally “under-performing” students.

National Center on Educational Outcomes

slide37

Usability

  • Universally designed assessments use text that enables people to read quickly, effortlessly and with understanding
  • The physical appearance of text – shapes of letters and numbers – conforms to several dimensions that characterize legible text

National Center on Educational Outcomes

slide38

OFFICIAL BALLOT, PALM BEACH COUNTY, FLORIDA

National Center on Educational Outcomes

slide39

Legible Text Dimensions

Contrast – Black type on matte pastel or off-white paper produces good contrast and reduces eye strain

Type Size – Print larger than 12 point increases legibility

Spacing – Space between letters and between words in wide

National Center on Educational Outcomes

slide40

Legible Text Dimensions

Leading – White space between lines of type (leading) is larger

Typeface – Standard typeface, with upper and lower case letters, is better than italic, small caps, or all caps

Justification – Unjustified text is easier to read, especially for poor readers

National Center on Educational Outcomes

slide41

Legible Graphs, Tables, Illustrations

  • Universally designed assessments use non-text materials just as carefully as text materials
  • Symbols are highly distinguishable
  • Only essential illustrations are used (ones referred to in text and necessary to answer question) [illustrations for interest often draw attention away from construct being assessed]

National Center on Educational Outcomes

slide42

Is the border distracting?

National Center on Educational Outcomes

slide43

Legible graphs, tables, illustrations

What’s that big black rectangle?

National Center on Educational Outcomes

slide44

Could this item be presented in an alternate format? Braille?

Is the high number of items on the map and long list of cities necessary to respond to this item?

“According to this weather page, which place is the warmest on December 28?”

If you were flying to Chicago the day this weather page was printed, what information could you learn for your trip from this page?

National Center on Educational Outcomes

slide45

Here is an example of an item that could more easily

be translated into an alternate format.

National Center on Educational Outcomes

slide46

Legible Response Formats

  • Universally designed assessments consider the design of the response venue as well as the assessment itself
  • Large bubbles that avoid most challenges of low vision or difficulty with fine motor skills
  • Consideration of age of students in selecting format (avoid separate answer sheets for younger students)

National Center on Educational Outcomes

slide47

More information?

NCEO Resources

  • Visit: http://education.umn.edu/nceo
  • or Search for NCEO
  • Web site includes:
    • Topic Introduction
    • Frequently Asked Questions
    • Online and Other Resources

National Center on Educational Outcomes

slide48

UD and Data Analysis

  • Goal = Increase validity for all
  • Focus = Reduce differential validity
  • Impact = May or may not reduce differential difficulty (p values)
  • Process = Go beyond internal validity statistics such as DIF (Differential Item Functioning)

National Center on Educational Outcomes

salvage example day 1
Salvage Example - Day 1

National Center on Educational Outcomes

salvage example day 2
Salvage Example - Day 2

National Center on Educational Outcomes

salvage in detail
Salvage In Detail

ND

SD

String

String

Pole

Pole

National Center on Educational Outcomes

keep it on the level

Keep It On the Level

Karen Barton

CTB/McGraw-Hill

National Center on Educational Outcomes

ud how do we know
UD – How do we know . . .
  • . . .if something is UD?
  • . . .if the UD is a valid and reliable approach for students?

National Center on Educational Outcomes

what are we looking for
What are we looking for?
  • Elements of UD
  • Content representation
  • Construct irrelevant barriers
  • Effect on student performance
  • Effect of accommodations

National Center on Educational Outcomes

check the design
Check the Design
  • Item reviews
    • UD elements
      • In place? What’s missing? What’s appropriate?
    • Content validation – test specs., content & standard rep.
    • By experts on various ability groups (SD, LEP, Gifted) and intended constructs
    • What about including students in the review?

National Center on Educational Outcomes

check the construct ion
Check the Construction
  • Construct validation
    • irrelevant variance, dimensionality
    • factor analyses, structure equation modeling, etc.

National Center on Educational Outcomes

check the effects pre vs post ud
Check the effects: Pre vs Post UD
  • For target groups:
    • What elements improve accessibility?
    • What elements decrease accessibility?
    • What elements have no effect?
  • If there is any effect, is it desired and feasible?

National Center on Educational Outcomes

check the affects pre vs post ud
Check the affects: Pre vs Post UD
  • Pilot Administration:
    • Student centered focus groups, “think-alouds,” interviews, questionnaires/surveys
  • Item Review
    • Accessibility expectations
      • possible impediments (linguistic loads, other elements not being implemented)
      • Amenable to accommodations
    • P-values by item, point biserials, etc.
    • DIF
      • Inferential - limited by sample sizes of subgroups
      • Descriptive - mean parameter values, objective score compares
    • Distracter analyses
    • Omit rates

National Center on Educational Outcomes

check the affects pre vs post ud59
Check the affects: Pre vs Post UD
  • Total score
    • Mean comparisons
    • Score changes – mean difference, effect size
    • External validation

National Center on Educational Outcomes

slide60
Is re-construction required?

(If it ain’t broke . .)

    • What are the stakes? Who is affected?
    • What are the costs?
      • To students
      • To contract
      • To test design
      • Time, money, experience

National Center on Educational Outcomes

plan ahead
Plan ahead!
  • RFP should request studies be conducted to assure the UD is being done, done correctly, and is a positive approach for improving the accessibility for students with diverse ability levels – BEFORE students receive high stakes consequences!

National Center on Educational Outcomes

slide62

Remember the Alamo

National Center on Educational Outcomes

ad