Defining Reading Proficiency for Accessible Large Scale Assessments

1. Defining Reading Proficiency for Accessible Large Scale Assessments Principles and Issues Paper American Educational Research Association April 10, 2006 Deborah R. Dillon, University of Minnesota & John Sabatini, ETS

2. A collaboration between two projects funded to conduct research and development on accessible reading assessments for students with disabilities that affect reading. Goals include producing research findings and assessment techniques that demonstrate how large-scale assessments can become more accessible and valid. Overview of the Project

3. Students deserve the opportunity to show their proficiency as readers – to show what they know and are able to do; this should be reflected in the way reading proficiency is defined. Expectations for students’ reading performance should not be lowered; rather, accommodations should be built into the test itself—via universal design—allowing participation of the widest range of students & more valid inferences about performance. Flexible expressions of reading will be necessary to allow students who may not be proficient on all components of reading to demonstrate the skills they do know. A Vision for Accessible Assessments for Students with Disabilities

4. First Task in the Project To design accessible reading assessments we need precise definitions of the constructs being measured. Step one involves developing a definition of “reading proficiency.” To accomplish this task we synthesized information and described: (a) a set of principles that will guide the research and development phases of our projects, and (b) unresolved issues that need to be addressed related to each principle.

5. Sources for the Development of the Three Principles and Related Issues • A review of existing definitions of reading proficiency (e.g., reports such as NRP, RAND, PISA, PIRLS) • A panel of experts to provide input • Focus groups

6. Principle 1: Definitions of reading proficiency must be consistent with core NCLB provisions. Principle 2: Reading proficiency must be defined in such a way that flexible expressions of reading are allowed while preserving the essential nature of reading. This is crucial as we seek to make assessments accessible to students with a variety of disabilities. Principle 3: Definitions of reading proficiency must reflect both comprehension and foundational skills. Overview of Principles

7. Principle 1: Definitions of reading proficiency must be consistent with core NCLB provisions. Accessible large-scale assessments developed by NARAP must adhere to two requirements of NCLB: “to provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading.” NCLB requires states to address areas of reading proficiency and component skills through grade-level definitions of content standards; NARAP cannot impose a particular standard—we must allow for the variability among states.

8. Access to, participation in, and progress in the general curriculum is a foundational requirement. The 2009 NAEP assessment, which is required to be used by all states, focuses on grade-level “cognitive targets,” and defines these as “the mental processes or kinds of thinking that underlie reading comprehension.” Assessments used primarily for accountability should also provide useful information to educators as they plan instructional improvements. Principle 1—cont.

9. How do the important reading skills vary as a function of grade level? How do we determine which measure may be appropriate for use at a specific grade level? How much can achievement levels vary and still meet the requirements of grade-level content? How are differences in reading achievement standards (e.g., modified or alternate achievement standards) developed and defined? How are these varying achievement standards reflected in definitions of reading proficiency? Issues Under Principle 1

10. Proficient readers, including students with disabilities, may rely on any set of component proficiencies to read and may compensate for some skills they lack by drawing on others—depending on their disability (e.g., a student with congenital deafness may comprehend what he/she reads, but need to deploy alternate strategies to understand sound-symbol relationships) Public views of what constitutes reading are reflected in various ways; these are indicated in how states allow/do not allow different approaches to assessments in reading and if assessments are allowed without restrictions Principle 2: Reading proficiency must be defined in such a way that flexible expressions of reading are allowed while preserving the essential nature of reading. This is crucial as we seek to make assessments accessible to students with a variety of disabilities.

11. Federal statutes allow a range of options in types of assessments used and the achievement standards applied to students with disabilities. Some students can take assessments based on modified achievement standards or alternate assessments based on grade level standards or on alternate achievement standards. Alternate achievement standards must be aligned to grade-level content standards but may differ from grade level achievement standards in breadth, depth, or complexity. Principle 2:--cont.

12. How broadly can we define what constitutes “reading” and still have the definition based on grade-level achievement standards? Can what constitutes as “reading” for standards-based assessments differ by disability category or by needed accommodation? When do the concepts of modified and alternate achievement standards apply to grade-level reading standards? How so students with disabilities compensate for weaknesses in specific reading proficiency components due to their disability or multiple disabilities? Issues under Principle 2

13. Panels of researchers and practitioners indicated that the NARAP core construct of a definition of reading should include both foundational reading skills and comprehension. NARAP is basing its work on the NCLB definition, which emphasizes foundational reading skills, and the 2009 NAEP framework, which stresses comprehension and indicates that students must apply foundational skills to comprehend a variety of texts for various purposes and situations. Principle 3: Definitions of reading proficiency must reflect both comprehension and foundational skills.

14. It could be argued that if a child demonstrates proficiency on an assessment focused on comprehension (e.g., 2009 NAEP), that no other assessment is necessary; conversely, foundational reading skills may not be developed or are in process for the population NARAP is studying. The overall goal is to develop flexible assessments to assess reading comprehension and/or foundational skills based on student performance. Principle 3—cont.

15. To what degree can component skills be measured independently? Comprehension is the primary goal for readers. If students are proficient in this area with accommodations, do we need to measure the foundational skills? If comprehension is our primary goal, should the comprehension score be weighted more heavily when foundational skills are also assessed? What feasible techniques are available for measuring foundational skills such as fluency or phonics knowledge in the context of large-scale assessment? Issues under Principle 3

16. For students who do not achieve grade-level proficiency, what processes can be developed or applied to aggregate their performance on component skills into an overall measure of reading proficiency? Some components appear to be problematic for certain disabilities (e.g. phonemes and deaf students). Do we develop alternate definitions of proficient reading for these populations based on a better understanding of reading processes and performances? Are some skills less critical to measure than others? Issues under Principle 3—cont.

17. If foundational skills were only assessed after a student had performed below proficient on comprehension, what proportion of ALL students (with and without disabilities) would be assessed on each of the foundational skills? Is this proportion small enough to assess students in small groups, individually, or via computer? Can some foundational skills be assessed together (e.g., decoding and phonemes)? If foundational skills are going to be measured only for students who are not proficient on an assessment of comprehension, can accommodations be allowed that invalidate the foundational skills (e.g., read aloud for decoding or extra time for fluency). If a student is not proficient on a measure of reading comprehension, should listening comprehension be assessed prior to measuring foundational skills? Issues under Principle 3—cont.

18. The three principles will guide NARAP in formulating the definitions of reading proficiency used within the studies. The definition of reading proficiency will be an organizing framework that supports states. Research will be conducted consistent with Principle 1 and addressing issues listed under Principles 2 & 3; several issues cannot be resolved through empirical research; these require information from NARAP to impact policies on assessing students with disabilities that impact reading. Conclusions

19. NARAP Web Site The Principles & Issues Paper is posted at: http://www.narap.info/