Welcome to the Race to the Top Assessment Program Technical Assistance Public Meeting

1. Welcome to the Race to the Top Assessment ProgramTechnical Assistance Public Meeting Creating Valid, Reliable, and Fair Assessments for Students with Disabilities & English Learners Washington, DC August 10, 2011 Please silence all cell phones and pagers. Thank you!

2. Race to the Top Assessment (RTTA) Program Overview and Meeting Goals Joe Conaty Patrick Rooney U.S. Department of Education

3. RTTA Public Meetings • This is the third in a series of public meetings on RTTA. • Two prior meetings: April 15 on State and Local Technology Infrastructure and June 10 on Automated Scoring of Assessments • Details on additional meetings will be forthcoming • Purpose of the meetings: • To provide technical assistance to and to support collaborative efforts of PARCC and SBAC as they develop new assessment systems • To expand the knowledge and expertise of the Department and the public around key assessment issues • To facilitate discussion of key components of the systems with experts and the public at large • Funded in part by The William and Flora Hewlett Foundation

4. RTTA Program Goals • Support states in delivering a system of more effective and instructionally useful assessments that: • Provide accurate information about what students know and can do by: • Eliciting complex student demonstrations or applications of knowledge and skills, as appropriate • Accurately measuring student achievement across the full performance continuum • Accurately measuring student growth over a full academic year or course; • Helping educators determine whether individual students are ready for college and careers by the time of high school graduation and, in previous grade levels, whether they are on-track for readiness • Reflect good instructional practice and support a culture of continuous improvement • Effectively assess all students, including students with disabilities and English learners

5. Looking Forward • Assessment systems must include one or more summative assessment components that are fully implemented by every state in each consortium by SY 2014-15, and are administered at least once during the academic year in, at a minimum: • Reading/language arts and mathematics • Grades 3-8 and high school • Results used to inform: • Teaching, learning, and program improvement • Determinations of school effectiveness • Determinations of principal and teacher effectiveness for the purposes of evaluation and support • Determinations of individual student college and career readiness

6. RTTA Grantees • Nearly $360 million awarded in September 2010 to two consortia, which together represent 45 states and DC: • Partnership for Assessment of Readiness of College and Careers (PARCC) • Project Management Partner: Achieve • SMARTER Balanced Assessment Consortium (SBAC) • Project Management Partner: WestEd • PARCC and SBAC have demonstrated commitment from institutions of higher education (IHEs) in member states that they will use the results of these assessments to determine entry into credit-bearing courses • The IHEs represent 90% (PARCC) and 74% (SBAC) of students who matriculate directly from K-12

7. Students with Disabilities & English Learners • The absolute priority required the consortia to create assessments for all students, including English learners and students with disabilities • The consortia are required to develop tests accessible for these populations and to create and standardize accommodations policies • Each consortium must develop a definition of “English learner” that is uniform across member states • Additional consortia: • Alternate assessments for students with the most significant cognitive disabilities • English language proficiency

8. Expectations for the Meeting • We have invited a range of experts to this meeting to share their knowledge and experience with the consortia members, looking at both current state of research and promising approaches to improving accessibility for these students • Format: • The morning will focus on key questions that need to be addressed regarding the needs of students with disabilities and English learners in the assessment system and possible methods to address those questions • The afternoon will focus on two standards – one in English language arts and one in mathematics – in a practical application of the issues to creating valid, reliable, and fair assessment items for these populations

9. Meeting Agenda 9:00-9:35 Welcome/setting the stage 9:35-10:15 Fishbowl discussion 10:15-10:30 Break 10:30-Noon Fishbowl discussion continued Noon-1:00 Lunch 1:00-1:30 Fishbowl discussion of public comments 1:30-3:00 Table exercise 3:00-3:15 Public comments 3:15-3:30 Wrap-up 3:30 Adjourn

10. Invited Experts • Jamal Abedi, University of California, Davis • Lizanne DeStefano, University of Illinois • Rebecca Kopriva, Wisconsin Center for Educational Research • Mike Russell, Measured Progress • Stephen Sireci, University of Massachusetts, Amherst • Guillermo Solano-Flores, University of Colorado, Boulder

11. Public Comments • ED wants to hear from the public on key considerations for creating valid, reliable, and fair assessments for students with disabilities and English learners • In the morning: • Comment cards are available at the registration desk • ED, the consortia, and the experts will discuss the comments/questions as time allows at the start of the afternoon session • All input from comment cards will be posted on our website • In the afternoon: • We have scheduled time for verbal public comment from 3:00-3:15 pm • Sign up to speak during the lunch break at the registration desk • Time limit: Up to 3 minutes per person/organization • Due to limited time, those not able to provide comments in person may email them to: racetothetop.assessment@ed.gov

12. Reminders • Please place all cell phones and other devices on vibrate • Race to the Top Assessment resources – Applications, FAQs, plus today’s materials and transcription available at: http://www2.ed.gov/programs/racetothetop-assessment • The purpose of this event is to promote a full discussion and hear a wide range of viewpoints on creating valid, reliable, and fair assessments for English learners and students with disabilities, as well as the challenges and opportunities afforded by the Race to the Top Assessment program. Through this meeting, the U.S. Department of Education is not seeking to promote and/or endorse any particular program, project, methodology or approach to this work.

13. INTRODUCTIONS Patrick Rooney U.S. Department of Education Meeting Facilitator

14. Accessibility and Accommodations Deborah Matthews - Kansas – Accessibility & Accommodations Workgroup dmatthews@ksde.org

15. The Purpose of the Consortium • To develop a set of comprehensive and innovative assessments for grades 3-8 and high school in English language arts and mathematics aligned to the Common Core State Standards • Students leave high school prepared for postsecondary success in college or a career through increased student learning and improved teaching • The assessments shall be operational across Consortium states in the 2014-15 school year

16. 29 Member States

17. Work Groups • Transition to Common Core State Standards • Technology Approach • Assessment Design: Item Development • Assessment Design: Performance Tasks • Assessment Design: Test Design • Assessment Design: Test Administration • Reporting • Formative Processes and Tools/Professional Development • Accessibility and Accommodations • Research and Evaluation

18. Accessibility And Accommodations Workgroup Purpose Ensure the SBAC Assessment System is maximally accessible to the broadest range of students through • identifying, recommending, and evaluating strategies, tools, and technologies, thereby • providing information and guidance that will positively impact critical aspects of assessment design and development

19. Accessibility and Accommodations Workgroup • New paradigm that focuses on the student first, not the test items which addresses accessibility issues as part of item development, not as an afterthought. • Computer based assessment allows technology to open many doors for students because accessibility is built into the assessments. • The necessity of accommodations is reduced. Accommodations that are allowed are more targeted.

20. Accessibility and Accommodations Workgroup In both policy and practice, SBAC will • include the broadest range of students • by facilitating each student’s ability to demonstrate as fully as possible what they know and can do • on the targeted constructs being measured • in a manner that is equitable and reliable, and yields valid interpretations of results.

21. Five Goals & Key Activities • Create policies that reflect current research, best practices, and future possibilities related to accessibility and accommodations • Create assessments that are free from bias and sensitivity issues leveraging new technologies, including interoperability while preserving test constructs • Create accessible and accommodated assessments that will yield valid and reliable results

22. Five Goals & Key Activities • Ensure accessibility and accommodations practice and policy are implemented with fidelity • Develop useful reporting and presentation guidelines that include information on accessibility and accommodations actions in the aggregate and at the individual student level

23. SBAC Representativeswww.smarterbalanced.org Michael Hock – Vermont – Accessibility & Accommodations Workgroup Co-Chair Michael.Hock@state.vt.us Wendy Carver - Utah – Accessibility & Accommodations Workgroup Wendy.Carver@schools.utah.gov ShelbiCole – Connecticut – Performance Tasks Workgroup Shelbi.Cole@ct.gov Gaye Fedorchak – New Hampshire – Accessibility & Accommodations Workgroup gfedorchak@ed.state.nh.us VijiSomasundaram – Wisconsin – Item Development Workgroup visalakshi.somasundaram@dpi.wi.gov

24. Accessibility for Students August 10, 2011 www.PARCConline.org

25. The PARCC Vision • Create high-quality assessments that measure the full range of the Common Core State Standards • Build a pathway to college and career readiness for all students and make accurate and reliable determinations as to whether students are “on track” or “ready” for college and careers • Provide information that supports various accountability uses (e.g., school, educator, student) • Provide timely and actionable information that supports continuous improvements in curriculum and instruction that inform effective classroom instruction and assessment practices • Leverage technology for a variety of uses: innovative items, accommodations, administration, and scoring and reporting. • Report results that allow for comparability across all PARCC states, across consortia, and to national and international assessments.

26. PARCC Accessibility Goals The Partnership will: Work to minimize/eliminate features that are irrelevant to what is being measured and measure the range of complexity of the standards so that students can demonstrate their knowledge; Design each component in a manner that allows ELL students and students with identified needs to demonstrate what they know and can do; Apply principles of universal design for accessible assessments throughout every stage of developing assessment components, items, and performance tasks; Leverage technology for delivering assessment components as widely accessible as possible; and Establish a Committee on Accessibility and Accommodations comprised of knowledgeable testing officials from member states (OWG).

27. PARCC Governance Structure Steering Committee

28. PARCC Technical Working Groups (TWG) Technical Advisory Committee (TAC) Leadership Team (LT) Technical Working Groups (TWG) Operational Working Groups (OWG) Limited number of groups convened by the TAC to address high priority topics that would benefit from collective problem-solving by leading experts Comprised of domain-specific technical advisors who interact with leadership and working groups and report to the TAC Accessibility, Accommodations, and Fairness TWG: Committee members represent a range of expertise in accessibility and accommodations Role is to help guide the efforts of working groups in designing accessible assessments that remain true to the intended constructs

29. Diane August Center for Applied Linguistics (ELL) David Edyburn University of Wisconsin-Milwaukee (SWD) Claudia Flowers University of North Carolina – Charlotte (SWD) Dianne Piche Leadership Conference on Civil Rights Charlene Rivera George Washington University (ELL) Diane Spence Region 4 Education Service Center, Braille Services (Braille) Martha Thurlow National Center on Educational Outcomes (SWD) Dan Wiener, Chair Massachusetts Department of Elementary and Secondary Education (accommodations for state assessments) Gerunda Hughes will serve as the liaison to the PARCC Technical Advisory Committee (TAC). Accessibility, Accommodations, and Fairness TWG Invited Members

30. PARCC Operational Working Groups (OWG) Technical Advisory Committee (TAC) Leadership Team (LT) • Who: Comprised of state representatives, Achieve staff members, and eventually vendor representatives • What: Responsible for the day-to-day aspects of work of key components of work • Why: To ensure efficient and effective collaboration among PARCC members to meet PARCC goals. Technical Working Groups (TWG) Operational Working Groups (OWG)

31. Roberta Alley (Chair / Leadership Team) Arizona Department of Education Trinell Bowman Maryland State Board of Education Mira Monroe Colorado Department of Education Melissa Fincher (Leadership Team) Georgia Department of Education Charity Flores Indiana Department of Education Andrew Hinkle Ohio Department of Education Leila Williams Arizona Department of Education Bambi Lockman Florida Department of Education Phyllis Lynch Rhode Island Department of Education Michael Reid Oklahoma State Department of Education Lori Rodriguez Florida Department of Education Dan Wiener Massachusetts Department of Elementary and Secondary Education Jessica Tickle Achieve/PARCC Danielle Griswold Achieve/PARCC Accessibility, Accommodations, and Fairness OWG Members

32. Accessibility and AccommodationsWorking Groups The Working Groups will be responsible for: • Drafting a set of Partnership-wide policies in a Partnership Accommodations Manual to be adopted by each member state for identifying eligible students, selecting allowable accommodations, and administering accommodations. That process will include: • Analyzing extant state accommodation policies, • Building a list of recommended standard accommodations, • Identifying constructs and research accommodations, • Recommending a set of proposed accommodation policies for the assessment, • Drafting a common Partnership Accommodations Manual, • Ensuring comparability in assessment administrations, • Monitoring ongoing refinements of accommodations, and • Developing training modules for IEP teams. Technical Working Groups (TWG) Operational Working Groups (OWG)

33. Accessibility and AccommodationsWorking Groups Technical Working Groups (TWG) Operational Working Groups (OWG) Adopting key policies and definitions that will include: • a common definition of “English Learner”; • a common set of policies and procedures for providing assessment accommodations for English learners and students with identified needs; and • a common set of policies and procedures for participation of English learners and students with identified needs in the assessment system.

34. Accessibility and AccommodationsWorking Groups Technical Working Groups (TWG) Operational Working Groups (OWG) Accessibility and Accommodations as a part of the development process • Design review and feedback • Test blueprint development • Technology development and selection • Passage and media review committee involvement • Item review committee involvement • Bias and sensitivity committee involvement • Testing the efficacy of assessment items with accommodations with the intended groups of students in pilot and field testing • Including sufficient number of students with identified needs (across sub-categories) in pilot and field testing • Data review committee involvement

35. Accessibility and AccommodationsWorking Groups Technical Working Groups (TWG) Operational Working Groups (OWG) • Build accessibility throughout the test itself with no trade-off between accessibility and validity • Use a combination of ‘accessible’-authoring and accessible technologies from the inception of items and tasks • Establish and maintain a close working connection with the Technology, Design, and Research Working Groups

36. UNDERSTANDING THE POPULATION Patrick Rooney U.S. Department of Education

37. Who are students with disabilities? Source: U.S. Department of Education, SY 2008-09 Annual Performance Reports. Figure courtesy of the National Center on Educational Outcomes

38. Understanding the Population • In 2008-09, some 6.5 million children ages 3-21 received special education services (13 percent of the population) • 95 percent were enrolled in regular public schools • 57 percent spent most of their time in general classes • The vast majority of students with disabilities take the general reading/language arts and mathematics assessments • Students with significant cognitive disabilities: Current law permits up to 1 percent of all students in the state (approximately 10 percent of students with disabilities) to take an alternate assessment based on alternate academic achievement standards • All students are expected to have access to, and be assessed against, grade-level content standards

39. Trends in Growth of English Learners Percentage of children ages 5-17 who spoke a language other than English at home and percentage who spoke a language other than English at home and spoke English with difficulty: Selected years, 1980-2009 Source: Census Bureau; NCES Condition of Education 2011

40. Where are English learners? Percentage of children ages 5-17 who spoke a language other than English at home and spoke English with difficulty, by state or jurisdiction: 2009 Source: Census Bureau; NCES Condition of Education 2011

41. Understanding the Population • Over 300 different languages are spoken by students. • Top 5 languages spoken by English learners: • Spanish (3.5 million) • Vietnamese (93,000) • Chinese (80,000) • Arabic (71,000) • Hmong (50,000) • In 2000, some 64 percent of English learners were born in the United States • 42 percent were 2nd generation • 22 percent were 3rd generation • The HHS Office of Refugee Resettlement identified 18,500 refugee children ages 6-18 in 2010.

42. UNDERSTANDING THE POPULATION What are the challenges or key questions that need to be addressed by the consortia when developing their assessment systems?

43. BREAK

44. A Brief Overview • Accessible Assessment & APIP • Michael Russell

45. A Two-Way Street • Accessibility & Assessment

46. A Two-Way Street • Accessibility & Assessment Test Item Access to Construct ITEM PresentContent Stimulate Construct ApplyConstruct Interact w/Content Visible Product ofConstruct ProduceResponse

47. What we can already do: • Digital Assessment Delivery • Match Content Form to Student Access Need • Language • Translation (Directions, Whole Item, Individual Words/Phrases) • Simplified English • Audio • Text-to-Speech, Pre-recorded Voice • Text-based content, Graphics & Tables, Non-Visual Descriptions • Braille • Sign

48. What we can already do: • Digital Assessment Delivery • Match Interaction & Response Modes to Need • Alternate keyboards • Tab-Enter control devices • Touch screens • Eye gaze • Speech-to-text

49. Interoperable Accessibility • Current Challenges • Standard Coding for Student Access Needs/Accommodations • Standard Tagging System for Item Content Accessibility Information • Standard File Exchange Format for Student Access Needs and Accessible Test Items

50. What is APIP? • Accessible Portable Item Profile Standard File Exchange Format Students Test Items Item Content Accessibility Information Meta Data Companion Materials Access Needs Profile