1 / 28

Show Me an Evidential Approach to Assessment Design

Show Me an Evidential Approach to Assessment Design. Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers. A Brief Introduction to Evidence Centered Design (ECD). Michael Rosenfeld, Ph.D. Rosenfeld & Associates. What is Evidence Centered Design?.

barbra
Download Presentation

Show Me an Evidential Approach to Assessment Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers

  2. A Brief Introduction to Evidence Centered Design (ECD) Michael Rosenfeld, Ph.D. Rosenfeld & Associates

  3. What is Evidence Centered Design? • A formal framework for assessment design • Emphasizes assessment as an evidential argument • Encourages models and representations that strengthen validity arguments • Focuses the test development process to better target decisions of interest

  4. The Basic Models of ECD • Proficiency Model – What you want to measure • Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies • Task Models – How to elicit valid and reliable evidence

  5. ECD Validity Chain Tasks Task Model Evidence Tasks Task Model Proficiency Evidence Tasks Evidence Task Model Tasks Task Model Tasks

  6. What is Gained from ECD? • Better understanding of the domain, for new constructs • Explicit chain of evidence from assessment goals, to models of evidence, to assessment tasks • Strong construct validity argument • Rapid and efficient item generation from models • automatic item generation • Opportunities for innovation, task types other than multiple choice • Philosophical shift from content-authoring to evidential argument

  7. Typical Test Development The Job Content Specification Job Analysis Major Dimensions Knowledge & Tasks TestBlueprint Incumbent Survey ItemWriting Content Linking Analysis Statistical Modeling CutScores

  8. Typical Test Development and Evidence Centered Design Typical Test Development Evidence Centered Design Content Specification Proficiency Model TestBlueprint Evidence Model ItemWriting Task Model Analysis Tasks CutScores

  9. Evidence Centered Design Claims (Criterion) Job Analysis Proficiency Model Reporting Variables Cognitive Task Analysis Knowledge, Strategies & Skills Evidence Model Evidential Argument Statistical Modeling Task Model Task Design Content Linking Tasks

  10. Still to Come • Example of the Proficiency Modeling process (Jay) • Illustration of Evidence Modeling and Task Modeling (David) • Commentary from an applied perspective (Barbara)

  11. Proficiency Models in the ECD Framework F. Jay Breyer, Ph.D.

  12. Proficiency Model • Combination of • Formalization of assessment goals, population and purpose • Formal claims to be made on the basis of the assessment • He/she is competent to operate an automobile • Proficiencies of interest • Knowledge of rules, procedures, and laws • Ability to implement driving regulations in practice • Skill in automobile operation

  13. What is a Proficiency? • Proficiencies are knowledge, skills and abilities that provide the basis for making claims about people

  14. Building a Proficiency Model: Assessment Goals • Assessment Goals • Protect the public health, safety & welfare • Examinee population • All eligible (age, prior training requirements, etc.) citizens self-selecting into the program • Test users/stakeholders • State regulatory bodies • State resident drivers • Other citizens exposed to driving • Result Usage/Decisions • State issuing the license (to issue or deny) • Means of official identification

  15. Building a Proficiency Model: Assessment Claims • Formal Claims are developed for assessment goals • Claims can be course-grained and general • Can safely operate a motor vehicle • These general claims can be be supported by a variety of more specific sub-claims in a hierarchy • Operates a vehicle in conformance with posted signs • Stops at a posted stop sign • Obeys posted speed limits • The final hierarchy of claims drives the evidential argument and the score reporting needs of the test

  16. Building a Proficiency Model: Proficiencies • Proficiency variables are the basis of making claims • Proficiency variables, like claims, may be organized hierarchically • Values or levels of proficiency variables may be selected to support claim distinctions • Informed by investigations of the domain and learning within the domain • job analyses, cognitive task analyses, learning models, etc.

  17. Example of a Hypothetical Proficiency Model for Driving

  18. Still to Come • Illustration of Evidence Modeling and Task Modeling (David) • Commentary from an applied perspective (Barbara)

  19. Providing the Evidence in Evidence Centered Design David M. Williamson, Ph.D.

  20. Building Evidence Models • Objective: valid inference • Understanding what constitutes evidence and why • Situations that elicit evidential behaviors • Task design • Statistical representation of evidential value of observations • Integration of scoring with design • ECD models – conceptual scoring

  21. Establishing Evidence in a “Perfect World” Targets of Inference Links behaviors and inference Behavior Elicitation • Driving Ability • Knowledge of RotR • Psychomotor • Evidence • Knowledge declarations • Performance • Task Requirements • Situation • Outcomes

  22. Establishing Evidence: Driver’s Exam • Driving Ability • Knowledge of RotR • Psychomotor • Intersections • Recognizing signs • Specifying intersection traffic behaviors • Following proper procedure • Knowledge of RotR • Weather & Light • Traffic Flow • Intersections • Speed • Psychomotor • Intersections • Speed • Turning • Parking

  23. Building Task Models: Why? • Elicits behaviors by predictably manipulating • Focus • Difficulty • Context • Targets what we want to measure • Provides a framework for generating multiple tasks targeting same proficiencies

  24. Building Task Models: What? To target elicitation of particular evidence, the task model specifies: • Tools provided • Directions and stimulus materials • Indication of data collected and retained for scoring • Relevance of specified elements to particular evidence, and particular proficiencies

  25. Evidence as Inference about Proficiency • Stage 1: Evidence Identification • Identify and summarize response elements • Stage 2: Evidence Accumulation • Using these elements to estimate ability

  26. Beneficial Outcomes of ECD • Development direction w evidential focus • Integration of expertise/activities (TD, stat, etc.) in the design process • Chain of reasoning • Integrated design – implications of design changes • Validity argument • Item modeling • AIG • Defensibility • Directed item writing – future forms • Somewhat more expensive (Initially)

  27. Still to Come • Commentary from an applied perspective (Barbara)

  28. Discussant Barbara Showers, Ph.D. Director, Office of Education and Examinations Wisconsin Department of Regulation and Licensing

More Related