innovation and technology in assessment comments
Skip this Video
Download Presentation
Innovation and Technology in Assessment: Comments

Loading in 2 Seconds...

play fullscreen
1 / 33

Innovation and Technology in Assessment: Comments - PowerPoint PPT Presentation

  • Uploaded on

Innovation and Technology in Assessment: Comments. Eva L. Baker Director National Center for Research on Evaluation, Standards, & Student Testing UCLA Public and Expert Input Meeting, Race to the Top, U.S. Department of Education Boston, MA November 12-13, 2009.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Innovation and Technology in Assessment: Comments' - etana

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
innovation and technology in assessment comments

Innovation and Technology in Assessment: Comments

Eva L. Baker


National Center for Research on Evaluation, Standards, & Student Testing


Public and Expert Input Meeting, Race to the Top, U.S. Department of Education

Boston, MA

November 12-13, 2009

© Regents of the University of California

overview big challenges underlying innovative tasks
Overview: BIG challenges underlying innovative tasks
  • Stimulate multi-level, multi-purpose system, open architecture, open source, scalable, top down-bottom up
  • Invest in design not just administration
  • Translate standards into rich representations of content, cognitive demands, situations and their relationships tuned for summative, interim, and formative uses
  • Design assessments to support learning
big challenges
Big Challenges
  • Obtain validity evidence, incentives for using new methods
  • Exploit new options, games, immersive environments, simulations only if necessary for assessment information, anticipate change, SCORM
  • Design to cost, with incentives for economies
  • Save $ using authoring systems, reusable templates and objects for continued renewal & amortization of system
  • Train users
  • Find mid point between page-turning and dazzle
1 criteria for innovative tasks other than efficiency
1. Criteria for Innovative Tasks(other than efficiency)
  • Emphasis on priority attributes of standard (s) in multi-step, realistic situations
  • Clear expectations
  • Teachable (or learnable), not trait-based
  • Opportunity for multiple paths to solution or product
  • Componential, reusable objects to permit quick creation of similar tasks
  • Tagged to key dimensions in database
  • Explicit feedback and areas for additional effort given
1 technology features off the shelf
1. Technology Features – off the shelf
  • Hand-held for place-based learning, networking, collaboration ,and teamwork, homework or informal assessment, feedback for teaching
  • Sensors to permit mapping of movement, engagement in tasks, screen time
  • Distraction, e.g., irrelevant animation
  • Monitor student activity, feedback, revision processes to guide instruction
  • Voice and handwritten inputs to broaden use
  • Potential rrelevant inferences
1 puppetman math problem solver
1. PuppetMan Math Problem-solver

© Regents of the University of California

2 3 4 focusing on first elements essential design components
2/3/4/ Focusing on first elements: essential design components

Graphical representation of standards

  • Content elements & relationships
  • Cognitive demands
  • Task situations
  • Linguistic requirements
  • Needed prior knowledge
  • Process or sequence of accomplishment
  • Generate relational database, add experience
2 3 4 mathematics ontology
2/3/4 Mathematics Ontology

© Regents of the University of California

2 3 4 algebra sequence
2/3/4 Algebra Sequence

© Regents of the University of California

2 4 algebra multiple units
2/4 Algebra Multiple Units

© Regents of the University of California

2 4 algebra part unit
2/4 Algebra Part Unit

© Regents of the University of California

2 3 4 cognitive readiness 21 st century skills transfer
2/3/4 Cognitive Readiness / 21st Century Skills: Transfer
  • Content expertise
  • Adaptive problem solving
  • Situation awareness
  • Decision making
  • Self-management (meta-cognition)
  • Teamwork
  • Communication
  • Resource access & management

© Regents of the University of California

2 3 4 cresst cognitive demands
2/3/4 CRESST Cognitive Demands



Situational Awareness

Risk Taking





Teamwork and




© Regents of the University of California

2 3 4 problem solving
2/3/4 Problem Solving

Problem Characteristics





Information attributes:






Actions & Strategies














Understanding/ Comprehension

Prior knowledge

Domain knowledge

Principles concepts





  • Solution Characteristics
  • Types (single/multiple)
  • Solution attributes:
  • - Adequacy
  • Availability
  • Appropriateness
2 3 4 key elements of assessment design development processes
2/3/4 Key Elements of Assessment Design & Development Processes
  • Vetted representations of knowledge and cognition-multi-purpose design
  • Ontology anchors database, supports coherence
  • Tasks based on models derived from research on learning, e.g., cognitive load, explanation, varied situations
  • Authoring systems and reusable task elements
  • Parsimonious, validated scoring guides exemplify expert performance
  • Transfer task item bank
2 4 assessment development platform
2/4 Assessment Development Platform
  • Collaborative space for users
  • Training for teacher-based scoring, including discourse, models, criterion levels for raters
  • Options for professional development for teachers and parents to assist in learning
  • Student instructional options or units
  • Expanding set of annotated examples of student work
  • Alternative automated approaches to scoring
  • Guidance and feedback for administration, including timing, requirements, privacy protection
3 4 summative assessment space
3/4 Summative Assessment Space
  • Standards and representations structure database
  • Collaborative space for designers
  • Reviewed for linguistic features, instructional sensitivity, fairness, accuracy
  • Data collection & analyses services
3 4 summative assessment space25
3/4 Summative Assessment Space
  • Meta-tags : representations/ situations/tasks/ student type/ unit/ sequences/criteria/ linguistics/ evidence by student
  • Planned accretion/pruning cycles -coherence of interim and formative systems
  • Validity prior to implementation
  • Cost targets & monitoring
  • Comparative benchmarks
3 4 summative assessment comments
3/4 Summative Assessment Comments
  • Technology-based systems afford integration of assessments used for interim or formative purposes in schools (with external) moderation to become part of a melded summative assessment
  • Platform needs to permit integrated assessments following expert review, data, and report reviews
  • Any summative assessment requires transfer tasks to avoid inappropriate teaching
summary transformation or incremental
Summary: Transformation or Incremental
  • Over the last 10 years, we have used tech-based measures interactively, over time, voice input, physical action sensors, collaborative
  • Built semi-automated design systems on cognitive demands and ontologies, object-oriented, fully queried reporting systems, formative assessments with micro instructional interventions; some have been brought to scale of 10 to 30% the going price
summary urgent evidence needs
Summary: Urgent Evidence Needs
  • New technical tactics to address extensive, adaptive performance ,e.g., comparability
  • Investment in small-sample statistical models to evaluate evidence before implementation
  • Incentives for alternatives to current psychometrics with evidence of longitudinal growth, value-added, plus
  • Achieved without compromising on the quality and fidelity of the measures and so that useful information is generated for teaching and learning
summary fairness cost
Summary: Fairness & Cost
  • Strategies for allowing student choice in assessment engagement
  • New approaches to measuring classroom processes with technology
  • Better privacy & security protections
  • New procurement policies
  • Please consider integrated assessment system
http www cse ucla edu

Eva L. Baker






[email protected]

© Regents of the University of California