Introduction to Accessible Reading Assessment June 14, 2008

1. CCSSO National Conference on Student Assessment Martha L. Thurlow Introduction to Accessible Reading Assessment June 14, 2008

2. Today’s Purpose • Highlight the challenges in reading assessment for students with disabilities • Provide an overview of relevant research design and analysis • Present research findings from projects funded to research and develop accessible reading assessments • Identify implications of the research for you • Share Principles and Guidelines of based on our research and other resources 2

3. National Accessible Reading Assessment Projects • Designing Accessible Reading Assessments (DARA) • Partnership for Accessible Reading Assessment (PARA) • Technology Assisted Reading Assessment (TARA) 3

4. NARAP Goals • Develop a definition of reading proficiency • Research the assessment of reading proficiency • Develop research-based principles and guidelines making large-scale reading assessments more accessible for students who have disabilities that affect reading • Develop and field trial a prototype reading assessment 4

5. Designing Accessible Reading Assessments (DARA) • Educational Testing Service (ETS) • Focuses on students with learning disabilities • Focuses on component approach to assessing reading skills. Primary focus are: • Word Recognition • Reading Fluency • Vocabulary Knowledge • Comprehension 5

6. Partnership for Accessible Reading Assessments (PARA) • Collaboration of National Center on Educational Outcomes and U of MN Department of Curriculum and Instruction, CRESST, U of CA Davis, and Westat • Focus on all disabilities that impact reading, particularly: • Learning disabilities • Speech or language impairments • Mental retardation • Deafness or hard of hearing 6

7. Technology Assisted Reading Assessment (TARA) • ETS, NCEO and Center for Applied Special Technology (CAST) • Focus on students with visual impairments • Focus on: • Examining the performance of operational ELA tests for students with visual impairments • Development of prototype Technology Assisted Reading Assessment • Inclusion of VI students in NARAP field test 7

8. Background • All projects focus on standards-based general assessments based on grade-level achievement standards – the regular assessment! • Not focused on alternate assessments based on alternate achievement standards • Not focused on alternate assessments based on modified achievement standards Still, work may sometimes be applicable to these too 8

9. Who We Are Martha Thurlow (PARA, TARA) Cara Cahalan Laitusis (DARA, TARA) Linda Cook (DARA, TARA) David O’Brien (PARA) Jamal Abedi (PARA) Discussant – Peggy Carr 9

10. Who Are You? 10

11. Plan for Today 1:00 – 2:30 Introduction Issues for Students with Disabilities Research Design and Analysis 2:30 – 2:45 BREAK 2:45 – 3:35 Identifying Less Accurately Measured Students Impact of Motivation and Engagement 3:35 – 3:50 BREAK 3:50 – 4:30 Segmented Reading Passages Principles and Guidelines 4:30 – 5:00 Peggy Carr, Discussant 11

12. Workshop Notebook • Notebook Tabs – Guide to Workshop Process • Agenda • Each Topic • Powerpoint presentation • Resource materials • Biographies for Presenters • Notepaper 12

13. Ground Rules Ask questions for clarification Interact with us! Take care of own needs 13

14. Contact Information Martha L. Thurlow University of Minnesota 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN 55455 THURL001@umn.edu 14

15. Issues in Assessing Reading of Students with Disabilities June 14, 2008 CCSSO National Conference on Student Assessment Martha L. Thurlow

16. Poor performance of students with disabilities is a big indicator that there are issues in assessing their reading performance Data from state reading assessments shows that this is so. . . . 2

17. Gaps in Performance on Reading Assessments Elementary School 3

18. Gaps in Performance on Reading Assessments Middle School 4

19. Gaps in Performance on Reading Assessments High School 5

20. Gaps exist in reading performance at all school levels: • They increase as the grade level increases • They vary by state, but the variability seems to be more a function of the difficulty of the test than its accessibility (states with the lowest and highest average scores for students with disabilities have smaller gaps – probably due to ceiling and floor effects) 6

21. Disabilities affect reading in many ways – we explored the ways in which disabilities may affect reading for 7 categories and developed a report about each: • Visual Impairments • Deaf or Hard of Hearing • Autism • Learning Disabilities • Mental Retardation • Speech or Language Impairments • Emotional or Behavioral Disabilities 7

22. Purpose To provide general information about specific disabilities and how they interact with reading, so that reading professionals and others who might contribute to the development of accessible reading assessments understand some of the challenges that need to be addressed. 8

23. Disclaimer Papers clearly state that the purpose is to begin a discussion of the issues surrounding reading and students with each disability. The papers were not intended to be comprehensive research reviews. We have clarified that the papers are for people who do not know the disabilities or for those who have not considered the interaction of disabilities with reading. 9

24. Overview Students Receiving Special Education Services 10

25. Reading and Students with Visual impairments • Most students with visual impairments are not blind. • Tactile (braille) and auditory methods of accessing text are most common. • Common classroom supports and accommodations may not be available for state assessment. 11

26. Reading and Students who are Deaf or Hard of Hearing • Age of onset of hearing loss and other factors shape educational and communication experiences. • Many communication forms (e.g., American Sign Language, Manually Coded English, lip reading); cochlear implants have raised new issues. • State assessment policies vary in whether they allow commonly used accommodations. 12

27. Reading and Students with Autism • Many students with Asperger Syndrome can decode words well, but may lack comprehension skills (Barnhill, 2004). • Students with autism may find it difficult to screen out distractions. • Accommodations are not as often designated toward this group. 13

28. Reading and Students with Specific Learning Disabilities • 90% of students with learning disabilities identify reading as their primary difficulty (President’s Commission on Excellence, 2003). • The read aloud accommodation is one of the most common and controversial accommodations provided for these students. 14

29. Reading and Students with Mental Retardation • Historically, educators often skipped academics (including reading) in favor of functional, social, or motor skills. • Despite wide variety of characteristics that can influence reading (poor short-term memory, low-level meta-cognition), reading skills can be mastered by many students with mental retardation. • Access to the general curriculum, broader accommodations, and alternate assessments are aspects of reading achievement for students with mental retardation. 15

30. Reading and Students with Speech or Language Impairments • Since reading is a language-based skill, students without strong language skills may be at-risk. • Accommodations for these students reflect reading strategies used with them – read aloud, assistive augmentative communication devices, and frequent breaks during assessment. 16

31. Reading and Students with Emotional or Behavioral Disorders • Students need to compensate for lack of attention, distractibility, etc. • Some accommodations that are needed are generally acceptable (breaks, quite room), while other are questionable (motivational prompts, calming music). 17

32. Accommodations Accommodations are more of an issue for reading assessments than for other content areas. This occurs because many students use those accommodations that may produce invalid scores (known as modifications in most places). 18

33. Percentages of Students Using Certain Accommodations Not all states publicly report on the use of accommodations, much less the specific accommodations or percentages of students. States that do include: Colorado North Carolina Based on 2005-06 data 19

34. The Challenge for Accountability Students who use accommodations that produce invalid scores – modifications – will now count as nonparticipants in the assessment. 20

35. Accessibility An accessible assessment is one that reveals the knowledge and skills of students whose characteristics create barriers to accurate measurement of these on traditional reading assessments • It measures the same knowledge and skills, at the same level • It may reduce the need for accommodations 21

36. Implications • It is important to understand the characteristics of all students taking assessments, including those that may affect performance but are not what is being measured (e.g., short term memory) • Disabilities do not are not the “cause” of poor performance – most students with disabilities can perform at levels comparable to their peers – if we make sure they get access to the curriculum, instruction, accommodations, and accessible assessments! 22

37. Implications • Accommodations are an important part of accessibility – more important now than ever before to address them (which can be incorporated into the assessment, which really produce invalid scores) • We need to explore innovative approaches to improving accessibility – things that in the end may benefit all students 23

38. Contact Information Martha L. Thurlow University of Minnesota 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN 55455 THURL001@umn.edu 24

39. Research Design and Analysis June 14, 2008 CCSSO National Conference on Student Assessment Cara Cahalan Laitusis and Linda Cook

40. Overview Types of Questions Research Designs and Analyses Case Example from DARA project Can read aloud and standard scores be reported on the same scale? Questions and answers 2

41. Research Questions Are test scores from accommodated and non-accommodated tests: Psychometrically comparable? Measuring the same construct? Equally valid predictors of the construct? What changes to test items (or administration) can: Increase/decrease accessibility? Increase/decrease validity of scores? Engage students with disabilities? Provide useful feedback to teachers? 3

42. Research Studies Opinion Research Item Tryouts Experimental Studies Analysis of Operational Test Data 4

43. Opinion Research Types of Opinion Research Surveys Interviews Focus Groups Potential uses: Explore the types of changes to test items (or administration) that may be worthy of additional research Identify problems in assessment design or administration 5

44. Item Tryouts Cognitive Labs (Think Alouds) Pilot Testing Field Testing 6

45. Item Tryouts Cognitive Labs (Think Alouds) 9-20 students per subgroup Requires one-on-one administration Qualitative analysis of responses 7

46. Item Tryouts Pilot Testing 20-40 students per subgroup Group administration Qualitative and Quantitative analysis of responses 8

47. Item Tryouts Field Testing 100 students per subgroup Group administration Quantitative analysis of responses 9

48. Experimental Studies Are test scores from accommodated and non-accommodated tests: Psychometrically comparable? Measuring the same construct? Equally valid predictors of the construct? What changes to test items (or administration) can: Increase/decrease accessibility? Increase/decrease validity of scores? Engage students with disabilities? Provide useful feedback to teachers? 10

49. Requirements of Experimental Studies Large sample sizes Random assignment of students to experimental groups to eliminate Form effects Order effects (test form or accommodation) Examinations of change generally require: Two samples (students with and without disabilities) Two testing conditions (standard and test change) Two equated test forms 11

50. Pros and Cons Experimental Design Pros Disentangle accommodation or test change from disability Impact of test change on total test score can be directly measured Impact of other effects (order, test form, disability-accommodation interactions) can be mitigated Cons Expensive Time consuming May not be able to simulate testing environment for high stakes testing 12