Cognitive Task Analysis for Teams

1. Cognitive Task Analysis for Teams Nancy J. Cooke New Mexico State University CTA Resource On-line Seminar October 11, 2002

2. Acknowledgements • NMSU Faculty: Peter Foltz • NMSU Post Doc: Brian Bell • NMSU Graduate Students: Janie DeJoode, Jamie Gorman, Preston Kiekel, Rebecca Keith, Melanie Martin, Harry Pedersen • US Positioning, LLC: Steven Shope • UCF: Eduardo Salas, Clint Bowers • Sponsors: Air Force Office of Scientific Research, Office of Naval Research, NASA Ames Research Center, Army Research Laboratory Nancy Cooke

3. Overview • What is team cognition? • Q&A • “Shared” mental models • Q&A • Holistic CTA for teams • Conclusions • Q&A Nancy Cooke

4. What is Team Cognition? Nancy Cooke

5. Team Cognition in Practice Nancy Cooke

6. Experimental Context CERTT (Cognitive Engineering Research on Team Tasks) Lab A Synthetic Task Environment for the Study of Team Cognition Five Participant Consoles Experimenter Console Nancy Cooke

7. Defining Team “…a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership” Salas, Dickinson, Converse, and Tannenbaum (1992) Nancy Cooke

8. Defining Team Cognition • It is more than the sum of the cognition of individual team members. • It emerges from the interplay of the individual cognition of each team member and team process behaviors Nancy Cooke

9. Team Cognition Framework Individual knowledge Team Process Behaviors Team Knowledge Team Performance Nancy Cooke

10. Team Cognition Framework Collective level + + Individual knowledge Team Process Behaviors Holistic Level Team Knowledge Team Performance Nancy Cooke

11. Team Knowledge • Long-term knowledge • Taskwork • Teamwork • Fleeting Knowledge (i.e., momentary understanding, situation model) • Taskwork • Teamwork Nancy Cooke

12. Measurement Limitations • Measures tend to assume homogeneous teams • Measures tend to target collective level • Aggregation methods are limited • Measures are needed that target the more dynamic and fleeting knowledge • Measures are needed that target different types of long-term team knowledge • A broader range of knowledge elicitation methods is needed • A need for streamlined and embedded measures • Newly developed measures require validation Nancy Cooke

13. Other Related Work • Group Think (Janis, 1972) • Distributed Cognition (Hutchins, 1991) • Common Ground in Discourse (Clark & Schaefer, 1987; Wilkes-Gibbs & Clark 1992 ) • Group Decision Support (Fulk, Schmitz, & Ryu, 1995) • Social Decision Schemes (Davis, 1973; Hinsz, 1999) • Transactive Memory (Wegner, 1986) • Shared Mental Models (Cannon-Bowers, Salas, & Converse, 1993) Nancy Cooke

14. Why Do We Care? • Outcome measures of team performance do not reveal why performance is effective or ineffective • Team cognition is assumed to contribute to team performance • Understanding the team cognition behind team performance should facilitate interventions (design, training, selection) to improve that performance Nancy Cooke

15. Team Cognition and Functions of Cognitive Task Analysis • Elicitation: Interviews, observations, think aloud used to make knowledge explicit • Assessment: Judgments are made regarding specific elicited knowledge (e.g., accuracy, intrateam similarity) • Diagnosis: Patterns in elicited knowledge (i.e. symptoms associated with dysfunctional or exceptional performance) are tied to a diagnosis Nancy Cooke

16. Questions or Comments? Nancy Cooke

17. Shared Mental Models Nancy Cooke

18. “Shared Mental Models” Shared Mental Models Shared Knowledge Nancy Cooke

19. “Shared” Sharing = to have compatible knowledge Sharing = to have the same knowledge “Shared beliefs” “Share the pie” vs. To hold in common To distribute Nancy Cooke

20. The “Apples and Oranges” Problem Measures to assess team knowledge often assume knowledge homogeneity among team members. • Shared knowledge = similar knowledge • Accuracy is relative to single referent Person A Person B Referent Nancy Cooke

21. Teams, by Definition, Consist of “Apples and Oranges” Airport Incident Command Center Telemedicine Nancy Cooke

22. “Shared” Knowledge Knowledge Base Person A Person B Shared = Common Nancy Cooke

23. “Shared” Knowledge Shared = Complementary Nancy Cooke

24. “Shared” Knowledge Shared = Common and Complementary Nancy Cooke

25. “Shared” Knowledge Common and Complementary Knowledge and Shared Perspectives/Varied Granularity Nancy Cooke

26. “Shared” Knowledge Conflicting Knowledge Irrelevant Knowledge No Coverage Common and Complementary Knowledge and Shared Perspectives Nancy Cooke

27. An Approach to the Apples and Oranges Problem Measures of team knowledge with heterogeneous accuracy metrics Nancy Cooke

28. Experimental Context • Five studies: Two different 3-person tasks: UAV (Uninhabited Air Vehicle) and Navy helicopter rescue-and-relief • Procedure: Training, several missions, knowledge measurement sessions • Manipulate: co-located vs. distributed environments, training regime, knowledge sharing capabilities, workload Nancy Cooke

29. Experimental Context MEASURES • Team performance: composite measure • Team process: observer ratings and critical incident checklist • Other: Communication (flow and audio records), video, computer events, leadership, demographic questions, working memory • Taskwork & Teamwork Knowledge, Situation Awareness Nancy Cooke

30. Long-term Taskwork Knowledge • Factual Tests • Psychological scaling The camera settings are determined by a) altitude, b) airspeed, c) light conditions, d) all of the above. How related is airspeed to restricted operating zone? Nancy Cooke

31. Long-term Teamwork Knowledge Given a specific task scenario, who passes what information to whom? Teamwork Checklist ___AVO gives airspeed info to PLO ___DEMPC gives waypoint restrictions to AVO ___PLO gives current position to AVO AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke

32. Team Situation Awareness • Assess accuracy and similarity of situation models of team members • SPAM (Situation Present Assessment Method) queries--display not interrupted • Queries about future events • Team members queried in random order at designated point in scenario within a 5-minute interval Durso, et al., 1998 How many targets are left to photograph? Nancy Cooke

33. Traditional Accuracy Metrics Team Referent .50 Team Member: Air Vehicle Operator 50% ACCURACY Nancy Cooke

34. Heterogeneous Accuracy Metrics AVO Referent DEMPC Referent PLO Referent Team Referent .33 1.0 0 .50 ACCURACY Overall: .50 Positional: 1.0 Interpositional: .17 Team Member: AVO AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke

35. Results Across Studies • Taskwork knowledge is predictive of team performance But… • True for psychological scaling, not factual tests • Timing of knowledge test is critical Nancy Cooke

36. Knowledge Profiles of Two Tasks Knowledge profile characterizing effective teams depends on task (UAV vs. Navy) Nancy Cooke

37. Knowledge Profiles of Two Tasks Complementary Common UAV Task Command-and-Control Interdependent Knowledge sharing Navy Helicopter Task Planning and execution Less interdependent Face-to-Face Nancy Cooke

38. Knowledge Acquisition Training Mission Experience Procedure: Taskwork Knowledge Knowledge Acquired: Teamwork Knowledge Teamwork knowledge is acquired through mission experience and its acquisition seems dependent on a foundation of taskwork knowledge acquired in training. Nancy Cooke

39. Results: Team Situation Awareness • Team SA mirrors the performance acquisition function and generally improves with mission experience • Team SA is generally good predictor of team performance (especially a repeated query) SA and Performance data from first UAV study. Nancy Cooke

40. Implications of Heterogeneous Metrics • Can deal with “apples and oranges” issue • Can assess knowledge underlying task performance • Knowledge profiles of tasks can inform training and design interventions Nancy Cooke

41. Future Directions on Apples and Oranges Problem • Apply metrics to fleeting knowledge • Embed knowledge measures in task • Need a taxonomy of tasks and additional profile work • Need to connect the knowledge profile (symptoms) to diagnosis of team dysfunction or excellence Nancy Cooke

42. Questions or Comments? Nancy Cooke

43. Holistic CTA for Teams Nancy Cooke

44. Team Cognition Framework Collective level + + Individual knowledge Team Process Behaviors Holistic Level Team Knowledge Team Performance Nancy Cooke

45. The “Sum of All Team Members” Problem Team Process Behaviors Team Knowledge Team Performance Individual knowledge Collective level + + The Problem: Measures are taken at the individual level and aggregated, as opposed to being taken at the holistic level. Holistic Level Nancy Cooke

46. The Sum of All Team Members Problem • Aggregating individual data is problematic given the apples and oranges problem • Team process behavior is missing from collective measures • Cognition at the holistic level should be more directly related to team performance Nancy Cooke

47. Our Approach to the Sum of All Team Members Problem • Consensus assessment tasks • Consensus concept ratings • Consensus teamwork checklist • Consensus SA queries • Communication as a measure of team cognition Nancy Cooke

48. Consensus Assessment Tasks An Example: Concept Ratings • Step One: Individual Concept Ratings collected • Present to each individual: • airspeed – altitude (1=related, 5=unrelated) • Responses: • AVO=4, PLO=1, DEMPC=5 • 2) Consensus Ratings Collected • Present to the team: • airspeed – altitude (1=related, 5=unrelated) • Prior responses: AVO=4, PLO=1, DEMPC=5 • Team discussion: PLO: “Well I said related since my camera settings for shutter speed and focus are dependent on each of these values” DEMPC: “OK, let’s go with that 1 it is” AVO= Air Vehicle Operator PLO = Payload Operator DEMPC = Navigator Nancy Cooke

49. Consensus Assessment Tasks Results • Consensus measures correlate moderately with performance compared to collective measures • Perhaps consensus does not adequately tap in-mission process behavior • Although collective measures and process behaviors predict team performance for co-located teams better than holistic measures, this is not true for distributed teams Nancy Cooke

50. Communication as a Window to Team Cognition The “Good” • Observable • Team behavior diagnostic of team performance • Think aloud “in the wild” • Reflects team cognition at the holistic level • Rich, multidimensional (amount, flow, speech acts, content) Nancy Cooke