Components of judgmental skill

1. Public Administration and Policy PAD634 Judgment and Decision Making Behavior Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany State University of New York T.STEWART@ALBANY.EDU

2. Fundamental tenet of correspondence research • "Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment." • Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282). components-of-skill.ppt

3. Y Y e s Brunswik's lens model Cues Distal variable Judgment X components-of-skill.ppt

4. True Cues Subjective Descriptors Cues Judgment Expanded lens model Distal variable components-of-skill.ppt

5. Components of skill and the lens model True Cues Subjective Descriptors Cues Distal variable Judgment Environmental predictability Reliability of information processing Fidelity of the information system Reliability of information acquisition Match between environment and judge components-of-skill.ppt 5

6. Alternative measures of performance(correspondence approach) • Area under ROC curve • Correlation (rYO) O is the observation, or “gold standard” Y is the judgment • Mean square error: • Skill score: components-of-skill.ppt

7. Decomposing accuracy--the skill score Murphy (1988): Skill = Correlation - Conditional bias - Unconditional bias score (regression bias) (base rate bias) components-of-skill.ppt

8. Decomposing accuracy: The Lens Model Equation (Tucker 1964) components-of-skill.ppt

10. Decomposition of skill score Skill score = components-of-skill.ppt

11. components-of-skill.ppt 11

12. Components of skill 1. Environmental predictability • Environmental predictability is conditional on current knowledge and information. It can be improved through research that results in improved information and improved understanding of environmental processes. • Environmental predictability determines an upper bound on performance and therefore indicates how much improvement is possible through attention to other components. components-of-skill.ppt

13. Environmental predictability limits accuracy of judgment components-of-skill.ppt 13

14. Components of skill 2. Fidelity of information system • Judgmental skill may be degraded if the information system that brings data to the judge does not accurately represent actual conditions, i.e., if the cues do not accurately measure the true descriptors. Fidelity of the information system refers to the quality, not the quantity, of information about the cues that are currently being used. • Fidelity is improved by developing better measures, e.g., though improved instrumentation or increased density in space or time. components-of-skill.ppt

15. Components of skill 3. Match between environment and judge • The match between the model of the judge and the environmental model is an estimate of the potential skill that the judge's current strategy could achieve if the environment were perfectly predictable (given the cues) and the judgments were unbiased and perfectly reliable. • This component might be called “knowledge.” It is addressed by training and experience. If the judge learns to rely on the most relevant information and ignore irrelevant information, this component will generally be good. components-of-skill.ppt

16. Components of skill Reliability • Reliability is high if identical conditions produce identical judgments. • Humans are rarely perfectly reliable. • There are two sources of unreliability: • Reliability of information acquisition • Reliability of information processing components-of-skill.ppt

17. Components of skill Reliability • Reliability decreases as amount of information increases. Theoretical relation between amount of information and accuracy of judgment components-of-skill.ppt

18. Components of skill Reliability decreases as environmental predictability decreases components-of-skill.ppt 18

19. Components of skill 4. Reliability of information acquisition • Reliability of information acquisition is the extent to which the judge can reliably interpret the objective cues. • It is improved by organizing and presenting information in a form that clearly emphasizes relevant information. components-of-skill.ppt

20. Components of skill Visual display of information Books by Edward R. Tufte TheVisual Display of Quantitative Information. Envisioning Information Visual Explanations Beautiful Evidence The Cognitive Style of Powerpoint: Pitching Out Corrupts Within Cheshire, Connecticut: Graphics Press components-of-skill.ppt

21. Components of skill Visual display of information Image from the videotape “Study of a Numerically modeled severe storm,” National Center for Supercomputing applications, University of Illinois (Tufte, Visual Explanations, p. 20) Tufte’s questions: How big is the cloud? What direction is it moving? What are the dimensions of the grid? components-of-skill.ppt

22. Components of skill Visual display of information Redesign by Tufte and Bushell (Tufte, Visual Explanations, p. 21) Restored quantitative information Directional arrows Grid size given (and reduce the dominance of the grid pattern) Time scale, with small clouds depicting storm history components-of-skill.ppt

23. Components of skill Visual display of information Chart prepared by Morton Thiokol for Challenger commission. Tufte, Visual Explanations, p. 47 Visual display problems • Disappearing legend. • The O-ring damage legend was on another slide. • Chartjunk • Obscures cause and effect • Temperatures turned sideways • O-ring anomalies depicted by scattered little marks • Wrong order • Date of launch rather than temperature components-of-skill.ppt

24. Components of skill Visual display of information: Scatterplot of temperature vs. O-ring damage Tufte, Visual Explanations, p. 45 components-of-skill.ppt

25. Components of skill Visual display of information influences mode of thought Use of images tends to induce intuition Use of numbers tends to induce analysis components-of-skill.ppt

26. Components of skill 5. Reliability of information processing • Decreases with increasing information and with increasing environmental uncertainty • Methods for improving reliability of information processing: • Limit the amount of information used in making judgments. Use a small number of very important cues. • Use mechanical methods to process information. • Combine several judgments. • Require justification of judgments. components-of-skill.ppt

27. Theoretical relation between amount of information and accuracy components-of-skill.ppt 27

28. The relation between information and accuracy depends on environmental uncertainty - - - - - Theoretical limit of accuracy ——— Actual accuracy - - - - - Theoretical limit of accuracy ——— Actual accuracy components-of-skill.ppt

29. Components of skill 6 and 7. Bias -- Conditional (regression bias) and unconditional (base rate bias) • Together, the two bias terms measure judgment "calibration.” • Reducing bias: • Experience • Statistical training • Feedback about nature of biases in judgment • Search for discrepant information • Statistical correction for bias components-of-skill.ppt

30. Calibration of judgments depends on the task Calibration data for precipitation forecasts (Murphy and Winkler, 1974) Heideman (1989) components-of-skill.ppt

31. Conclusion • Problem: Improving judgmental accuracy • Understanding and improving judgment requires understanding the task and the environment. • Decomposing skill can aid in identifying the factors that limit judgmental accuracy. components-of-skill.ppt