PARA Project Overview, Results, Next Steps Martha Thurlow and Deborah Dillon

1. PARA ProjectOverview, Results, Next StepsMartha Thurlow and Deborah Dillon

2. Overview: PARA Partners • Partnership for Accessible Reading Assessment (PARA) consists of: • University of Minnesota - NCEO (National Center on Educational Outcomes) and Department of Curriculum and Instruction • CRESST (Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles) and University of California, Davis • Westat

3. PARA Research Program Assumptions • Assumption #1: We don’t know everything about what goes into accessible reading assessment yet. • Assumption #2: Preliminary research must inform design of “testlets.” • Assumption #3: Both preliminary and experimental research will inform our knowledge of Principles and Guidelines (Goal 3) for future assessments.

4. Three Major Research Question Areas • What characteristics of students require more accessible assessment of reading? • What characteristics of current assessment practices hinder accessibility? • What characteristics would accessible assessment have?

5. PARA Activities and Findings • Disability Reports • Differential Item Functioning Analysis • Differential Distractor Functioning Analysis • Test Characteristics Study • Student Characteristics Pilot Study

6. Disability Reports* Addressing disabilities that seem to affect reading . . . . . some of these are the focus of our projects and some are not • Visual Impairments • Deaf or Hard of Hearing • Autism • Learning Disabilities • Mental Retardation • Speech or Language Impairments • Emotional or Behavioral Disabilities * GAC members were instrumental in reviewing/writing these reports.

7. Differential Item Functioning (DIF) Analysis • Used to examine differences between students with and without disabilities – to see whether items functioned differently. • Found that many items exhibited DIF, especially for grade 9 students with disabilities, most often in the second half of the assessment. • Several issues with the data limited findings – NRT, no access to category of disability or accommodations information; concern about high omission rate.

8. Differential Distractor Functioning (DDF) Analysis • Used to examine patterns of incorrect answers to distractors – to see whether items functioned differently. • Found that many items exhibited DDF for students with disabilities in grade 9, especially in the second half of the assessment. • Several issues with the data also may limit these findings – NRT, no access to category of disability, accommodations information; concern about high omission rate.

9. Test Characteristics Study • Examined test specifications for purposes and constructs, number of items assigned to constructs, and types of items typically found. • Found that not all standards in states are assessed; yet,more than one-third of states assessed some form of foundational skills; further, in most states, comprehension and analysis/interpretation are considered an extension of foundational skills. • Not all states posted their test specs/blueprints on Websites; these states are being pursued now to complete the analysis.

10. Student Characteristics Pilot Study • Identifying students whose teachers believe have reading skills that are not accurately measured by state reading assessments. • Found that teachers are able to identify students, but did not identify many with comprehension limitations that obscured other reading skills and those who have strengths outside of what most reading tests cover. • Issues with follow-up measures made it difficult to verify teachers’ beliefs about skills.

11. PARA’s Next Steps • Chunking Study • Student Characteristics Study • Motivation Study

12. Chunking Study

13. Purpose of Study To examine the effects of chunking reading passages on the performance of students with disabilities, and to compare this effect to the effect on the performance of students without disabilities.

14. Definition Chunking refers to the concept of building breaks directly into the test booklet by separating the passage into more manageable “chunks,” and adding relevant test questions directly beneath the chunk.

15. Participants and Passages Participants: Grade 8 students with and without disabilities Passages: Three passages with multiple choice items. All were released state assessment items, for which permission to use was obtained

16. Design Students with and without disabilities will be randomly assigned to either Version A (standard version) or Version B (chunked version). All students will be given background questions, feedback question related to fatigue and mood, and a student motivation scale.

17. Student Characteristics Study

18. Purpose of Study To identify students whose teachers believe have reading skills that are not accurately measured by state reading assessments, and follow-up with students. To check on the prevalence of “less accurately measured students” (LAMS) with various characteristics.

19. Types of Students • Fluency limitations obscure comprehension skills • Comprehension limitations obscure other reading skills • Strengths exist outside of what most reading tests cover • Responds poorly to standardized testing circumstances or materials

20. Research Questions • How well can teachers identify LAMS? • How well can teachers document their rationale? • How well can a brief examination confirm or disconfirm teacher judgments?

21. Procedures • Teachers complete nomination questionnaire • Researchers interview teachers • Researchers interview students • Researchers examine students Students are 4th and 8th grade students with and without disabilities

22. Examination Protocols After interview, student examined with: • CBM, listen to text with immediate retelling • CBM, read text with immediate retelling • CBM, choice to read or listen to text, and do standard multiple choice questions • CBM, think aloud

23. Prevalence Study Teachers complete survey about all of their LAMS, indicating which of the types they have in their classroom and their characteristics

24. Designing Large Scale Reading Assessments that Increase Students’ Motivation during the Assessment

25. Purpose of Study To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement.

26. Research Questions 1. Does the option of exerting choice in the selection of passages on a large-scale reading assessment improve the accessibility of the assessment and lead to increased comprehension due to improved motivation and engagement for students with disabilities and general education students? 2. When students with disabilities and general education students are given the option to choose literary and informational passages that are included on large-scale reading assessments, do they perform better on one text type or the other?

27. Research Questions – cont. • 3. Are students who are identified as being more motivated to read on general motivation to read measures (e.g., scales and questionnaires) more proficient in reading when given a choice of reading passages on large-scale assessments? • 4. Are students who are identified as being more situationally motivated to read, as reported on questions following each passage, more proficient in reading those passages?

28. Participants & Setting • Participants: Students will be selected from grades 4 and 8. Students with disabilities as well as general education students will be in the experimental condition; comparable students will be in the control condition. • Setting: Computer lab/classroom, untimed testing situation, appropriate accommodations for students with various disabilities.

29. Design In the counterbalanced design of the study we are looking to see if there is increased performance between students (both general education and those with disabilities) who are given choice (C) in selecting the passages they read and those who are not (NC). A stratified random assignment of students will be necessary to ensure that students representing particular disabilities are randomly assigned to the experimental and control condition. Students will take the version of the test they are assigned.

30. Design—cont. • Students in the experimental conditions will be allowed to “design your own assessment.” • The assessments will include 4 total passages(two literacy-fiction and two informational-expository—based on the NAEP 2009 Reading Framework). • Separate pools of passages will be provided for 4th and 8th grades to select from; each passage will be followed by 5-6 multiple choice items.

31. “Choice Condition” • Students in the “choice” condition will be presented with short (1-2 sentence) descriptions/summaries for 6 literary-fiction passages. Students choose 2 of the 6 passages to include in their individually designed assessment. • Students will be presented with short (1-2 sentence) descriptions or summaries for 6 informational-expository passages. Students choose 2 of these 6 passages to include in their individually designed assessment. Students will be given time to read the descriptions prior to the initiation of the test. • After selecting the 4 total passages, students will be asked to briefly state why they chose these passages.

32. “Control Condition” • “Control” condition students will read 2 literacy-fiction and 2 informational-expository passages from the same high interest pool of passages, but they will not be allowed to choose the particular passages that will be included in their assessment. • “Control” condition students will be randomly assigned passages(Note. These passages will have been ranked as low interest by 4th and 8th grade students during a pre-calibration study).

33. “Control Condition” • “Control condition” students will follow the same procedures as the experimental conditions students: Students will be presented with short (1-2 sentence) descriptions or summaries for 6 literary-fiction passages and 6 informational-expository passages. Students will be asked to rate these passages according to level of interest.

34. Passage Selection Criteria • Passages used in this assessment will be selected from existing literature and textbooks based on broad appeal to students at the 4th and 8th grades, respectively. There will be separate pools of passages for 4th and 8th grades. • Children’s literature scholars and the research literature will be consulted to select high interest passages.

35. Passage Selection Criteria—cont. • Reading passages will be chosen to represent two text types and one specific form of text within the text types: informational—exposition; & literary—fiction. • 18-24 high interest passages at the 4th grade level and the same number at the 8th grade level will be secured; half of the passages will be literary and half will be informational texts. • Short (1-2 sentence summaries will be written for each passage).

36. Passage Selection Criteria—cont. • To determine the difficulty of the passages for 4th and 8th grade readers, each passage will be rated as equivalent on factors such as difficulty and accessibility (using multiple reading ability measures) and structure (e.g., text grammar or top-level structure). • Permissions will be procured for passages; more passages will be secured than needed on the actual test and more items for each passage will be written (e.g., due to the calibration process and removal of weak items).

37. Skills Assessed: The Use of Cognitive Targets • Item writers will use the 2009 NAEP cognitive targets to create 10 items to follow each of the 18-24 passages for each grade level. • The tasks or questions students answer after reading passages will match the content and key ideas of individual passages rather than being generic questions.

38. Attending to Issues of Motivation • Texts and items will be ordered from easy to difficult to foster motivation via improved self efficacy. • Situated motivation questions will be woven into the test booklets for the choice condition only and will be placed after the items that go with each question; these questions will indicate students’ perceptions of the texts they read (e.g., difficulty; interest) and their sense of self-efficacy in reading and completing the items following the passage (the task). • A pre-assessment general motivation survey will be given to all students to provide information on student feelings about “self as reader” (e.g., Motivation for Reading Questionnaire-MRQ).

39. Demographic data on all students will be collected, including whether they are students with a disability and/or students who are English language learners. • Observational data and verbal protocols will be obtained from students to determine their perspectives on assessment instruments, tasks, and contexts.