GAAP: Guidelines for Accessible Assessments Project

1. GAAP: Guidelines for Accessible Assessments Project June 26th, 2014 CCSSO National Conference on Student Assessment New Orleans, LA The content of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education and you should not assume endorsement by the Federal government.

2. GAAP Project Background and Overview – Jen Higgins Research Design and Randomized Control Trial Findings – Lisa Famularo Cognitive Lab Findings – Vitaliy Shyyan Finalizing Audio and Sign Guidelines – Lisa Famularo Discussion, State and Consortia Perspective – Trinell Bowman and Deborah Matthews Session Agenda

3. Item Development Appropriate complexity of content, language, diagrams, and layout Delivery Interface Intuitive, non-distracting, input device independent, flexible tool support Universal Design Sets the Table for Access Universal Design

4. Delivery System Accessibility Supports Ability to concentrate on the problem is one of the constituent parts of the successful mental work

5. Delivery System Accessibility Supports We all see things the same way. We see words in groups or phrases.The print is more dominant than thebackground. The print shows nomovement. The printed letters areevenly black. Black print on whitepaper gives the best contrast foreveryone. White background lookswhite.

6. Traditional image of a test item. What is a Test Item?

7. Digital Item File ASL tags Braille Tags Spoken (Read Aloud)Tags Text-Based Default Form Translation Language Learner Tags

8. Focus is to develop research based guidelines for the audio and sign representation of test items. Primary project activities: Review of literature and existing state guidelines Draft consensus based audio and sign guidelines Conduct consensus building webex sessions for audio guideline issues Conduct in-person meetings for sign guideline issues Conduct cognitive labs Conduct randomized control trial Revise and finalize guidelines Project Overview

9. State Audio Guidelines for Assessment

10. State Audio Guidelines for Assessment • States (and consortia) are making progress in improving reliability of audio accessibility • Seven states provide oral/reader scripts that human administrators use to read tests to one or more eligible students. • Eleven states provide pre-recorded versions of test content on CD or cassette that accompany paper-based test materials. • Three states provide a pre-programmed audio version of the test as part of their computer based test administration. • Smarter Balanced and PARCC have developed audio guidelines that are being used to create accessible audio versions of CCSS test content.

11. Framework for Audio Guidelines • Project Background and Methodology • Intended Use • Item Example • Strategies for Graphic and Image Description • Use of Guidelines with Text-to-Speech • Content Specific Guidelines with Examples

12. Audio Guideline Example Parentheses Example: 3(x + y) = 6 Audio Guideline Text Only/Text and Graphics a) Read the parentheses by referring to the opening and closing of the parentheses using the language “open parenthesis” and “close parenthesis.” b) It is important to reference the close of the parentheses to be clear on when the parenthetical expression ends. c) When reading an equation or expression with multiple parts and sets of parentheses, pause to help differentiate between sections. d) Read brackets using the same language as parentheses, but with the word bracket (“open bracket” and “close bracket”). Application of Audio Guideline Example 1: Three times open parenthesis x plus y close parenthesis equals six.

13. Graphic and Image Description

14. State Sign Guidelines for Assessment • No states have content specific sign guidelines for assessment. Alaska, Massachusetts, and South Carolina provide a pre-recorded sign DVD for one or more grade levels/content areas. • Existing state documents provide guidelines for: • Qualification of sign language administrators • Warnings against cueing and elaboration • Use of “home signs” or “locally developed signs” • Testing environment (lighting, group size, etc).

15. Framework for Sign Guidelines • Introduction: rationale for guidelines, intended use • Team: roles and responsibilities in developing videos • Process: steps in creating high quality ASL videos • ASL Grammar: review of ASL grammar considerations to create high quality videos • Content Specific Guidelines (fingerspelling, equations, graphics and images, etc). • Filming Considerations

16. ASL Video Example

17. Identified areas of focus for the research during guideline development Cognitive labs to understand extent to which audio and sign representations developed according to the guidelines remove construct irrelevant barriers for students Randomized control trials (RCT) to understand impact on student performance Research

18. RCT Research Areas: Sign

19. RCT Research Areas: Audio

20. RCT Sample Sizes

21. State Participation in GAAP Note: The bolded states participated in the cognitive labs. The bolded states with an asterisk participated in both the cognitive lab and randomized trial.

22. RCT Design

23. Did general education students perform higher on the research test forms than students with sign and audio access needs? Yes. ASL Study (p<.001) Audio Study (p<.001) Randomized Control Trial Findings

24. Did students with audio and sign needs performed better on supported items than they did on unsupported items? Yes. ASL Study (p=.013) Audio Study (p=.025) Randomized Control Trial Findings

25. Individual Support Areas studied did not show statistically significant differences in student performance ASL: fingerspelling, use of space, plurality, equations, introducing an item Audio: graphs, equations, drag and drop, images, decimals Implications for Guidelines – purposefully use FS, equations, and incorporate graph/image description as linguistically appropriate Randomized Control Trial Findings

26. Cognitive Lab Research Areas • Grades 3 - 5 • Sign: use of space, perspective, equations, fingerspelling, plurality, numerals in answers • Audio: graphics and images, drag and drop, expressions/equations, numerals in answers • Grades 6 – 8 and Grades 9 – 12 • Sign: graphics and images, expressions/equations, fingerspelling, diamond (item set-up) • Audio: graphics and images, drag and drop, expressions/equations, numerals, decimals

27. Cognitive Lab Research Methodology • Students were asked to complete 4-5 item pairs using a computer based testing system with embedded audio or sign support. • Each item pair focused on one audio or sign representation issue. • Students were asked to explain any difficulties they had answering the item, any parts of the audio/sign representation they found confusing or helpful, and were asked to report which audio/sign representation they preferred

31. Student Demographics: Need

32. Student Preferences for Audio: Equations

33. Student Preferences for Audio: Diagrams and Images

34. Insights from Students: Audio • I liked how it actually told me what the equation was. (317) • The read aloud accommodation is helpful sometimes because sometimes I can’t really focus on what I’m reading. (210) • [equations read aloud] You know better the problem – you can think about it faster if the computer tells you the problem. (112) • [when shapes are described] You don’t have to think that hard. It helped me answer better. (106)

35. Student Preferences for Sign: Finger Spelling

36. Student Preferences for Sign: Diamond Presentation

37. Insights from Students: Sign • Some of the English words I don’t know so I can watch the sign. (262) • I can understand questions better when they are signed, not finger-spelled. (156) • ASL is our language. English is not bad but ASL is my language. … When you read the English, it is hard to understand. But the signs help. (167)