1 / 18

Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis

Rita Bush, Office of the Director of National Intelligence This presentation was given at the 2017 Serious Play Conference, hosted by the George Mason University - Virginia Serious Play Institute. This presentation describes a four-year, multi-team experimental research program designed to study the effectiveness of games as a training tool for teaching about and mitigating cognitive bias. Game designs were iterated over multiple development cycles, informed by the results of both playtesting and formal experiments. The research showed that it is possible to reduce biased decision-making both immediately and long-term.

SeriousPlayConf
Download Presentation

Rita Bush - How to Use Games to Mitigate Cognitive Bias in Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How to Use Games to Mitigate Cognitive Bias in Analysis Dr. Rita M. Bush INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  2. Office of the Director of National Intelligence Central Intelligence Agency Defense Intelligence Agency Department of State National Security Agency National Geospatial-Intelligence Agency Department of Energy Department of the Treasury National Reconnaissance Office Drug Enforcement Administration Army Federal Bureau of Investigation Navy Department of Homeland Security Air Force Coast Guard Marine Corps 3 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  3. IARPA Mission and Method To envision and lead high risk/high payoff research that delivers innovative technology for future overwhelming intelligence advantage. • Bring the best minds to bear on our problems – Full and open competition to the greatest possible extent – World-class, rotational, Program Managers • Define and execute research programs that: – Have goals that are clear, measureable, ambitious and credible – Employ independent and rigorous Test & Evaluation – Involve IC partners from inception to finish – Run from three to five years 4 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  4. Sirius Goal Use serious games to train analysts to recognize and mitigate their cognitive biases 5 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  5. Why Games? • Serious Games provide – Experiential learning – Freedom to fail – Repetition, repetition, repetition • Game/simulation based training is common in military, business, medicine, etc. • New analysts have likely grown up playing videogames 6 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  6. Cognitive Bias • Decision making relies on simple, fast, heuristics – “rules of thumb” • Heuristics gone astray = Cognitive Bias • Affects all decision-makers: medical professionals, law enforcement officers, intelligence analysts… you, me… • Examples in the IC – The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction (2005) – Lessons Not Learned: Postmortems on Intelligence Failures, 1950-2004, Kent Center for Analytic Tradecraft (2005) 7 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  7. Biases Examined by Sirius Program 8 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  8. Why Conduct this Research? • Cognitive biases are very hard to overcome • Studies of game-based learning treat the game as a “black box” • Limited research on game-based learning effects over time • No other research using games to mitigate cognitive biases • Very few successful examples of developing valid tests of cognitive bias 9 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  9. Example Screen Shots SUNY Albany “CYCLES” Leidos “Missing” 10 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  10. Assessment of Biases in Cognition (ABC) Instrument • Developed by ETS, an industry leader in assessment development and validation • Web-based assessment measuring susceptibility to cognitive biases • Extensive literature review available 11 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  11. Independent Validation & Verification (IV&V) Research Design (Johns Hopkins APL) Best & Final Versions of Performer Games ETS/MITRE Immediate Posttest ETS/MITRE Delayed Online Posttest Phase 1: 8 weeks Phase 2: 12 weeks Participants ETS/MITRE Pretest Random Assignment Demographics ETS/MITRE Standardized Measures of Cognitive Bias and Bias Knowledge ETS/MITRE Standardized Measures of Cognitive Bias and Bias Knowledge ETS/MITRE Standardized Measures of Cognitive Bias and Bias Knowledge IARPA Supplied IARPA Supplied Training Video Training Video 12 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  12. Sirius Answers 3 Questions Do games work as training tools? If yes, how much will the training effect decay, 8-12 weeks after final game-play? Do games work as well as a video? 13 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  13. All games showed significant improvement in biasmitigation scores The video shows the expected decay after 8 weeks Sirius Phase 1 game 8-week posttest scores stayed the same or improved! 14 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  14. Phase 2 Student Study Results All games & video improved bias mitigation from pre-test to post-test across biases – larger improvements for Missing & Cycles. Game bias mitigation dropped at 12 weeks but still improved over pre-test 15 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  15. Active vs Passive Learning (N = 305) Playing the game was better than watching the game, which was better than watching the video 16 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  16. Myth-Busting • Myth: Games only work well for gamers, or younger analysts, or males – “Non-gamers” learned just as much – No age or gender differences • Myth: High fidelity (close match to physical or organizational features of the job) is important – Simple graphics and game style worked just as well (or better) – “Cognitive fidelity” did matter 17 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  17. Point of Contact • Sirius Program Manager – Dr. Rita Bush – rita.bush@iarpa.gov • For more information, see: – www.iarpa.gov 18 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

  18. Thank You! 19 INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

More Related