Thoughts on the Design of an Assessment System for the “Race to the Top”

1. Thoughts onthe Design of anAssessment Systemfor the“Race to the Top” Jim Pellegrino

2. Issues & Challenges • Elements of a Comprehensive System • Going beyond Standards: Integrating Assessment with Curriculum & Instruction • Design Options for K-8 and High School • Capitalizing on Affordances of Technology • Science an Equal Partner • Phasing in Components & Building Capacity • Logic Models and a Theory of Action • Articulating RttT with ESEA & other Policies

3. 1. Elements of aComprehensive System • Desired end product is a multilevel system • Each level fullfills a clear set of functions and has a clear set of intended users of the assessment information • The assessment tools are designed to serve the intended purpose • Formative, summative or accountability • Design is optimized for function served • The levels are articulated and conceptually coherent • They share the same underlying concept of what the targets of learning are at a given grade level and what the evidence of attainment should be. • They provide information at a “grain size” and on the “time scale” appropriate for translation into action.

4. Multilevel Assessment System An Integrated System Coordinated across levels Unified by common learning goals Synchronized by unifying progress variables

5. Elements of aComprehensive System (cont) • The system is designed to track progress over time • At the individual student level • At the aggregrate group level • The system uses tasks, tools, and technologies appropriate to the desired inferences about student achievement • Don’t force everything into a fixed testing/task model • Use performance tasks, portfolios, projects, fixed- and open-response tasks as needed

6. 2. Integrate Assessmentwith Curriculum & Instruction • Assessment is part of a coordinated system that includes explicit assumptions about Curriculum and Instruction • All three components work in concert and are designed as complements • They are designed together rather than separately • Curriculum specifies what is to be learned and understood and why • Instruction specifies the features of learning environments -- what teachers, students and others do -- that are designed to promote the desired forms of learning • Assessment specifies how the processes and products of teaching and learning will be appraised relative to what was intended for outcomes

7. Assessment Should not be the “Tail that Wags the Dog” Assessment

8. Integrate Assessment with Curriculum & Instruction (cont) • Common Core Standards are a start but not enough • Standards are ambiguous and lack detail -- they need to be translated using “backwards design” and “evidence-centered design” processes • Need to be clear about the Claims one wishes to make about students, the Evidence that would back up those claims, and the Tasks that can provide the critical forms of evidence • There are good examples for how to engage in this ECD process at scale • Redesign of the AP science courses and exams • College Board science standards for college success

9. Using an Explicit Domain Modelof Knowledge and Skillsto Coordinate C-I-A in AP

10. 3. Consider Alternative Design Options for K-8 & High School • K-8 should be designed to prepare students for success in high school • Need to have a clear, coherent, and cumulative model linking C-I-A in each major subject-matter area. • The conceptual model needs to incorporate learning progressions and benchmark targets at critical age/grade levels • Assessment practices need to promote high levels of performance (and not undermine them) • Task characteristics should be matched to desired intellectual performance demands • Multiple assessment options may maximize information yield for differing intended levels of use • Mixture of census testing and matrix sampling may be optimal • Consider periodic vs yearly assessments in core subjects

11. Consider Alternative Design Options for K-8 & HS (cont) • High School should be designed to prepare students for college or work • The same model of testing as K-8 may be inappropriate and sub-optimal - not all students take same courses at same time • Alternatives need to be considered such as assessment closely linked to curriculum and instruction frameworks in core subject-matter areas • Options for certification of college-readiness and rapid transition to college should be considered in lieu of current “time in school” graduation models • Need flexibility rather than rigidity in considering possible transitions from high school to college or high school to vocational/occupational training

12. 4. Capitalize on the Multiple Affordances of Technology • Makes it possible to tap a broader repertoire of cognitive skills and knowledge • Can go beyond conventional practices for item presentation and conventional stimuli • Can implement a range of task designs and item formats • Can record and score complex aspects of behavior • Functioning models of technology use exist for NAEP and PISA -- basis from which to extrapolate • Can adapt assessments to the needs of different learners and student populations • May better enable use of Universal Design principles • Can implement adaptive assessment methods

15. Capitalize on the Multiple Affordances of Technology (cont) • Makes possible the embedding of assessments in learning environments • Permits the design and implementation of diagnostic and formative assessment systems • There already exist a number of workable systems for use in schools in areas of literacy, math and science • Technology can fulfill multiple functions in the design delivery, and use of assessments in multilevel systems that involve multiple users and multiple information needs

16. Examples of FormativeAssessment Tools

17. Multiple Functions of Technologyin an Assessment System

18. 5. Make Science anEqual Partner • Science at K-8 is in deperate need of improvement • The U.S. has no coherent model for what is to be taught and learned in K-8 science • International assessments show that science is one of our weakest areas and it gets worse across time • Leaving science out of the system contributes to its second tier status • Allows it to be treated as a “periodic” rather than ongoing and critical instructional issue • Science offers multiple ways to support the development of verbal and quantitative literacies • Much of science is about the process of reasoning from evidence -- doing so requires both qualitative and quantitative reasoning skills

19. Make Science anEqual Partner (cont) • In a 21st century global environment it makes little sense to ignore the scientific and technical literacy of our students • Many of our choices as citizens involve understanding the relationship between local, national and global actions and consequences • Science achievement at the high school level suffers from the poor preparation of students at K-8 • There is little in the way of carryover and cumulative knowledge about science content or process from K-8. • Need to reconsider what HS science needs to be taught and learned given advances in science and technology

20. 6. Phase in Components & Build System Capacity • Consider System Design & Implementation as a 7 to 10 Year Comprehensive Effort and Long-term Investment • Can’t build and implement all the components we need in a short time frame • Capacity to design and implement coordinated systems and quality assessments is not there -- not at the scale of design and implementation needed • Identify key leverage points over different time horizons • 1-3 years; 3-5 years; 5-7+ years • If USED rushes states to “do it all” the result may well be no significant improvement • Unintended consequence: set assessment back another 10 years like what happened with the mandates from NCLB

21. Phase in Components & Build System Capacity • Much of the change in the productive use of assessment will require extended training in the use of new tools and systems • A substantial professional development effort will be needed across levels of the system • Teachers, pirncipals, district and state leadership • Teacher education will need to adapt • Processes for the effective collection and use of assessment information will need to be implemented • New technologies and data systems will need to be created and then accommodated in the business practices of education • The shift from an audit mode for assessment to an assistance mode will take time, effort, and $$$$

22. 7. System Logic Model& Theory of Action • States should be asked to lay out a “Logic Model” for system design, implementation and evaluation • Logic Model forces attention to: (a) existing conditions, (b) resources, (c) inputs, (d) outputs, and (e) proximal and distal outcomes • Not only is there a focus on the elements of the system but most especially the assumed logical and causal relationships amnong them • Model enables monitoring the building of the system and its enactment • Model also enables strategies for evaluation of the system along the way and for adjustment and correction as needed

24. System Logic Model& Theory of Action • A “Logic Model” forces one to specify the presumed “Theory of Action” • Helps make explicit assumptions about how particular components are supposed to work, who is to be impacted, what the expected consequences should be, and WHY. • In a complex assessment system, it is critical that the theory of action be articulated, especially with regard to how assessment information is to be used to improve outcomes over time - who will use what information and how • Competing theories of action can be made explicit in the system design phase -- choices can be made based on the quality of the evidence and/or argument in favor of adopting one theory in lieu of alternatives

25. 8. Policy Articulation:RttT & ESEA • RttT involves a number of policy choices and it will have multiple implications regarding federal, state, and local regulations and funding • Be clear what the policy issues are -- consider the intended as well as potential unintended consequences of the RFP and awards process as well as all the regulatory elements once implemented • Listen carefully to the advice of “unbiased” groups like the NRC Board on Testing and Assessment on issues of assessment design and use, including the use of Value Added Methodologies for teacher evaluation • Figure out how RttT should align with likely changes in ESEA once reauthorized • Avoid the two having major conceptual and operational conflicts such that ESEA renders the RttT investment moot!

26. Issues & Challenges Redux • Focus on the design elements needed for creating a comprehensive and coherent assessment system • Integrate assessment with curriculum & instruction: Go beyond standards to unpack the connections • Consider purposes and design options for K-8 and High School C-I-A systems • Capitalize on the multiple affordances of technology • Make science an equal partner in the RttT effort • Phase in components & build human capacity • Require states to specify a system Logic Model with an accompanying Theory of Action • Think through the articulation of RttT with ESEA & other education and assessment policies