1 / 33

Innovation and Technology in Assessment: Comments

Innovation and Technology in Assessment: Comments. Eva L. Baker Director National Center for Research on Evaluation, Standards, & Student Testing UCLA Public and Expert Input Meeting, Race to the Top, U.S. Department of Education Boston, MA November 12-13, 2009.

etana
Download Presentation

Innovation and Technology in Assessment: Comments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Innovation and Technology in Assessment: Comments Eva L. Baker Director National Center for Research on Evaluation, Standards, & Student Testing UCLA Public and Expert Input Meeting, Race to the Top, U.S. Department of Education Boston, MA November 12-13, 2009 © Regents of the University of California

  2. Overview: BIG challenges underlying innovative tasks • Stimulate multi-level, multi-purpose system, open architecture, open source, scalable, top down-bottom up • Invest in design not just administration • Translate standards into rich representations of content, cognitive demands, situations and their relationships tuned for summative, interim, and formative uses • Design assessments to support learning

  3. Big Challenges • Obtain validity evidence, incentives for using new methods • Exploit new options, games, immersive environments, simulations only if necessary for assessment information, anticipate change, SCORM • Design to cost, with incentives for economies • Save $ using authoring systems, reusable templates and objects for continued renewal & amortization of system • Train users • Find mid point between page-turning and dazzle

  4. 1. Criteria for Innovative Tasks(other than efficiency) • Emphasis on priority attributes of standard (s) in multi-step, realistic situations • Clear expectations • Teachable (or learnable), not trait-based • Opportunity for multiple paths to solution or product • Componential, reusable objects to permit quick creation of similar tasks • Tagged to key dimensions in database • Explicit feedback and areas for additional effort given

  5. 1. Technology Features – off the shelf • Hand-held for place-based learning, networking, collaboration ,and teamwork, homework or informal assessment, feedback for teaching • Sensors to permit mapping of movement, engagement in tasks, screen time • Distraction, e.g., irrelevant animation • Monitor student activity, feedback, revision processes to guide instruction • Voice and handwritten inputs to broaden use • Potential rrelevant inferences

  6. K Mapper Student Work

  7. 1. K Mapper Student Action

  8. 1. K Mapper Expert Scoring Model

  9. 1. Rescuing Animals Using Ratios and Equations

  10. 1. PuppetMan Math Problem-solver © Regents of the University of California

  11. 2/3/4/ Focusing on first elements: essential design components Graphical representation of standards • Content elements & relationships • Cognitive demands • Task situations • Linguistic requirements • Needed prior knowledge • Process or sequence of accomplishment • Generate relational database, add experience

  12. 2/3/4 Mathematics Ontology © Regents of the University of California

  13. 2/3/4 Algebra Sequence © Regents of the University of California

  14. 2/4 Algebra Multiple Units © Regents of the University of California

  15. 2/4 Algebra Part Unit © Regents of the University of California

  16. © Regents of the University of California RILS - 23

  17. 2/3/4 Cognitive Readiness / 21st Century Skills: Transfer • Content expertise • Adaptive problem solving • Situation awareness • Decision making • Self-management (meta-cognition) • Teamwork • Communication • Resource access & management static.flickr.com/80/230668852_055b631d8c_b.jpg © Regents of the University of California

  18. 2/3/4 CRESST Cognitive Demands Content Understanding Situational Awareness Risk Taking Learning Adaptive Problem Solving Teamwork and Collaboration Communication Metacognition © Regents of the University of California

  19. 2/3/4 Problem Solving Problem Characteristics Givens Parameters Goals Constraints Information attributes: -Credibility -Relevancy -Certainty -Coherence -Ambiguity Actions & Strategies Evaluate Seek Identify (Re)define Navigate Explore Plan Design Monitor Revise Solve Select Understand Understanding/ Comprehension Prior knowledge Domain knowledge Principles concepts Skills Abilities Schema Procedures • Solution Characteristics • Types (single/multiple) • Solution attributes: • - Adequacy • Availability • Appropriateness

  20. 2/3/4 Problem solving representation

  21. 2/3/4 Key Elements of Assessment Design & Development Processes • Vetted representations of knowledge and cognition-multi-purpose design • Ontology anchors database, supports coherence • Tasks based on models derived from research on learning, e.g., cognitive load, explanation, varied situations • Authoring systems and reusable task elements • Parsimonious, validated scoring guides exemplify expert performance • Transfer task item bank

  22. 2/4 Assessment Development Platform • Collaborative space for users • Training for teacher-based scoring, including discourse, models, criterion levels for raters • Options for professional development for teachers and parents to assist in learning • Student instructional options or units • Expanding set of annotated examples of student work • Alternative automated approaches to scoring • Guidance and feedback for administration, including timing, requirements, privacy protection

  23. 3/4 Summative Assessment Space • Standards and representations structure database • Collaborative space for designers • Reviewed for linguistic features, instructional sensitivity, fairness, accuracy • Data collection & analyses services

  24. 3/4 Summative Assessment Space • Meta-tags : representations/ situations/tasks/ student type/ unit/ sequences/criteria/ linguistics/ evidence by student • Planned accretion/pruning cycles -coherence of interim and formative systems • Validity prior to implementation • Cost targets & monitoring • Comparative benchmarks

  25. 7th Grade Math Standards California CA

  26. 7th Grade Math Standards: Comparing AZ, MA, CA AZ MA CA

  27. Multi-purpose Assessment Development Platform Architecture

  28. 3/4 Summative Assessment Comments • Technology-based systems afford integration of assessments used for interim or formative purposes in schools (with external) moderation to become part of a melded summative assessment • Platform needs to permit integrated assessments following expert review, data, and report reviews • Any summative assessment requires transfer tasks to avoid inappropriate teaching

  29. Summary: Transformation or Incremental • Over the last 10 years, we have used tech-based measures interactively, over time, voice input, physical action sensors, collaborative • Built semi-automated design systems on cognitive demands and ontologies, object-oriented, fully queried reporting systems, formative assessments with micro instructional interventions; some have been brought to scale of 10 to 30% the going price

  30. Summary: Urgent Evidence Needs • New technical tactics to address extensive, adaptive performance ,e.g., comparability • Investment in small-sample statistical models to evaluate evidence before implementation • Incentives for alternatives to current psychometrics with evidence of longitudinal growth, value-added, plus • Achieved without compromising on the quality and fidelity of the measures and so that useful information is generated for teaching and learning

  31. Summary: Fairness & Cost • Strategies for allowing student choice in assessment engagement • New approaches to measuring classroom processes with technology • Better privacy & security protections • New procurement policies • Please consider integrated assessment system

  32. http://www.cse.ucla.edu Eva L. Baker voice fax email 310.206.1530 310.267.0152 baker@cse.ucla.edu © Regents of the University of California

More Related