1 / 75

The Human Performance Assessment and Modeling Laboratory

The Human Performance Assessment and Modeling Laboratory. Director: Ling Rothrock, Ph.D. Ph.D. Students: Damodar Bhandarkar Jeonghwan Jin Sungsoon Park Dongmin Shin Hari Thiruvengada Jing Ying. Overview.

albert
Download Presentation

The Human Performance Assessment and Modeling Laboratory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Human Performance Assessment and Modeling Laboratory Director: Ling Rothrock, Ph.D. Ph.D. Students: Damodar Bhandarkar Jeonghwan Jin Sungsoon Park Dongmin Shin Hari Thiruvengada Jing Ying Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  2. Overview Research within HPAM form a mutually-supporting triad with human-in-the-loop simulations informing the development of inductive learning models and testing hypotheses grounded in cognitive science Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  3. Investigating Team Performance Using a Human-in-the-loop Discrete-Event Simulation SystemSponsor: US NavyStudents: Hari Thiruvengada, Damodar Bhandarkar Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  4. Architecture Overview of Components Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  5. Roles Performance Individual Metric A Weapons Officer Tactical Action Officer Commanding Officer Avg Presence Avg Avg Hard Hard Individual Metric A Hard Software Agents Easy Team Performance Easy Easy HITL Test Bed TASP Telerobotic Agents Individual Metric A Human Operators Phones Consoles Face-to- Face Interaction User Model Scenario Pool Scenarios Research Focus • Using Human-in-the-loop Simulation Test bed (The Team Aegis Simulation Platform) to Investigate Individual and Team Performance Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  6. Models Overview of Research Class Editor Script/ Scenario Dictate class types and capabilities Wizard of OZ Enforces task consistency Expands experimenter control Hypothesis and Experimental Design Implements events to test hypotheses ScriptMaker Team Aegis Simulation Platform Dictates task consistencies Informs investigation / promotes follow-on hypotheses Operator performance profiles Enables performance assessment along defined metrics Data logging includes operator interaction, system states, and required activities Relational Database Informs model building Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  7. Experimentation Strategy Script Maker Scenario Phase 1: Scenario Generation Phase 2: Run Simulation TASP Simulation: Each member can communicate with the other using speech and internal messaging system Team Role A Team Role B Team Role C Output files C Output files A Output files B Phase 3: Analyze Team/Individual Performance Team/ Individual member performance Integrated Output files Performance Analyzer Tool Research Phases Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  8. Electronic Officer Workstation in TASP Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  9. Performance Measurement Individual and Team Time Windows Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  10. State of the world Signal Noise Response Hit False Alarm Detected Not Detected Miss Correct Rejection True State of Environment Perceived State of Environment Signal Detection Theory (SDT) • Provides ability to measure operator sensitivity • Provides means to capture conservative or risky behavior • Enables experimenter to set response criterion based on payoff structures Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  11. State of the world Task Demand No Task Demand Response False Alarm Early On-time Late Action No Action Correct Incorrect Miss Correct Rejection Time Windows and SDT • Incorporates temporal dimension • Consistent with research findings that suggests the primacy of perception & pattern recognition in dynamic DM Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  12. How It All Fits Together Consider an example: • Operator is an air traffic controller on the border between two nations • Operator instructions are to identify all unknown aircraft within the airspace and establish radio contact with all aircraft that come within range. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  13. How It All Fits TogetherSequence Diagram Representation Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  14. 50nm 40nm 30nm 20nm 10nm Test Bed Rules Rules of Engagement – 1. Engage track if W/I 20nm 2. Illuminate track if w/i 30nm 3. Issue level 1 warning if w/i 50nm 4. Issue level 2 warning if w/i 40nm 5. Issue level 3 warning if w/i 30nm 6. CAP no further than 256nm 7. CAP no closer than 20nm 8. Assign track primary ID to all unknown 9. Assign track designation to all unknown WASP Task: Operator – anti-air warfare coordinator Objective – ensure safety of ownship & monitor airspace surrounding ownship Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  15. A Visualization Framework for Bounding Physical Activities – Towards a Quantification of Gibsonian-Based Fields Students: Hari Thiruvengada, Jeonghwan Jin Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  16. Contents ■ Affordance Concept■ Research Focus■ Problem Statement■ Openings and Obstacles■ Mathematical Modeling■ Analysis of Possible Action Strategies■ Summary Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  17. Affordance Concept(1/4) ■ Affordance What the environment offers, provides or furnishes to the animal, either for good or ill (Gibson, 1979/1986).■ Multiple Formal Definitions - Affordances are relations between an animal and its environment that have consequences for behavior (Warren, 1984)- Affordances are properties of the environment of an animal that have consequences for the animal’s behavior (Turvey, 1992)- Affordances are emergent properties of the animal-environment system (Stoffregen, 2003) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  18. Affordance Concept(2/4) ■ Research areas in which affordance concepts (e.g., ecological optics and direct perception) are applied - Robotics Obstacle avoidance using optic flow (Ducheon, Warren et al., 1998) Reactive system using ecological approach (Murphy, 2000), Optic flow-based control of walking biped mechanism (Lewis, 2002) - Human Computer Interaction (HCI)User Interface Design (Norman, 1988; Gaver, 1991), Planning and User Interface Affordances (Amant, 1999) - Planning and Scheduling Path Finding (Raubal, 2000) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  19. ■Affordances for locomotion - Obstacle is a rigid object, detached or attached, a surface with occluding edges – it affords collision - Opening is an aperture, hole or gap in a surface, also with occluding edges – it affords passage ■Fields of safe travel - Field of possible paths which the car may take unimpeded - Boundaries of fields are determined by objects or features of the terrain with negative or positive valence in perception (obstacles)■Clearance lines - An object that impedes locomotion forms a negative valence in the space immediately around it in the actor’s field - This space is represented by lines of clearance Affordance Concept(3/4) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  20. (from Reed and Jones 1982, p.125) ■ Vehicle Locomotion Affordance Concept(4/4) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  21. ■ Develop a mathematical modeling of Gibsonian-based fields as a visualization framework for supervising communications-based activities between a mobile robot and an astronaut in order to; - represent a mobile robot and an astronaut’s possible action strategies to maintain their network communication in a continuous and dynamic environment, and - graphically represent bounds on actualized actions strategies of the robot and the astronaut based on their possible actions strategies Research Focus Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  22. Problem Statement • Consider a network communication system which consists of three elements: • Habitat (H) in which the supervisor works, • Mobile robot (R), and • Astronaut (A). • H is used as a mission control center responsible for supervising R and A. • R and A can move around to achieve their different scientific goals. • To communicate, R serves as a relay to maintain network communication for A. • In case of communications failure, R and A both must move back to one of their previous positions where they had confirmed their communication. • R and A would have multiple possible actions that they can take while satisfying the time constraint. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  23. NR4 NR5 NA2 A NA6 NA5 Mount NR6 NR3 NA3 R NR2 NA4 NA1 H Mount Mount NR1 Mount Graphical Representation Notations H: Habitat R: Mobile robotA: AstronautNRi: Network relay point for RNAi: Network relay point for A Operational Details •H is used to as a mission control center (a mission supervisor is residing in H).•R and A both have their own paths to follow when they travel.•R and A serve as a relay such that R is responsible for maintaining network communication for A.•R and A must check their network communication regularly (i.e., every 30 sec).• In case of communication failure between R and A, R and A both must move back to one of their previous NRi and NAi respectively where they had confirmed their communication. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  24. ■ In terms of restoring network communication betweenR and A - Openings: Network relay points, NRi and NAi, satisfying the following two conditions; 1. R and A’s network communication was confirmed at those points 2. R and A can go back to those points within a given time - Obstacles: Network relay points, NRi and NAi, not satisfying those two conditions mentioned above. Openings and Obstacles Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  25. Mathematical Modeling(1/2) ■ Basic Assumptions•R and A’s network relay points are always given in pairs (i.e., every NRi has a corresponding NAi and vice versa); • Average velocity of R, VR, is 80ft/min and average velocity of A, VA, is120ft/min;• At any point in time, current positions of R and A are known;• Distance between R’s current location and NRi, DRi, is known;• Distance between A’s current location and NAi, DAi, is also known;•R and A have found that their network communication was lost at NR6 and NA6 respectively; and• The duration of time, T, given for R and A to move back to (NRi, NAi) is 3.5 min. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  26. Mathematical Modeling(2/2) Step 1. Calculate Shortest Time (ST) to restore R and A’s communication with the assumptions that restoring communication between R and A is critical. ST = Min{Max(tRi, tAi)} for i = 1, 2, … , n. Eq.(1) , wheretRiis the minimum time required for R to move back to NRi (tRi=DRi/VR),tAiis the minimum time required for A to move back to NAi (tAi=DAi/VA), and n is the number of network relay points that R and A have confirmed their communication before their communication fails.In essence, Max(tRi, tAi) in Equation 1 is used to determine the longer time between tRi and tAi for each i and Min{Max (tRi, tAi)} in Equation (1) is used to select the shortest time among all possible longer times for all i. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  27. Mathematical Example Given Information: n = 5, DRi =(DR1, DR2, DR3, DR4, DR5) =(350ft, 290ft, 260ft, 220ft, 250ft), tRi =(tR1, tR2, tR3, tR4, tR5) (∵ tRi= DRi/ VR) =(4.375min, 3.625min, 3.250min, 2.750min, 3.125min) DAi= (DA1, DA2, DA3, DA4, DA5) = (400ft, 250ft, 340ft, 300ft,270ft), tAi = (tA1, tA2, tA3, tA4, tA5) (∵ tAi= DAi/ VA) = (3.333min, 2.083min, 2.833min, 2.500min,2.250min) ST = Min{Max[(tRi, tAi)]}, for i = 1, 2, …, 5. = Min{Max[(tR1,tA1), (tR2,tA2), (tR3,tA3), (tR4,tA4), (tR5, tA5)]} = Min{Max[(4.375, 3.333), (3.625, 2.083), (3.25, 2.833), (2.75, 2.5), (3.125, 2.25)]} = Min{4.375, 3.625, 3.25, 2.75, 3.125} = 2.75 min Since 2.75 min came from calculating tR4and NA4 is the corresponding network relay point for NR4, it specifies; 1. R and A must move back to NR4 and NA4 respectively to restore their communication as soon as possible and it will take at least 2.75 min to do that. 2. Since it will take only 2.5 min for A to get NA4, A will have 0.25 min(=15 sec) as extra time that s/he can utilize for her/his tasks before moving back to, on her/his way to, or after arriving at NA4. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  28. Analysis of Possible Action Strategies(1/5) Step 2. Identify R and A’s possible action strategies based on a given time, T, 3.5 min.•Possible (NRi, NAi) pairs: (NR3, NA3), (NR4, NA4), (NR5, NA5)• Time to travel to those pairs: (tR3, tA3), (tR4, tA4), (tR5, tA5) =(3.250, 2.833), (2.750, 2.500), (3.125, 2.250)Step 3. Calculate R and A’s extra times for the possible pairs of NRi and NAi identified in Step 2 using Eq. (2) and (3).ERi = T - tRi Eq.(2) EAi = T - tAi Eq.(3)For example, ER4=T - tR4=3.5 min – 2.75 min=0.75 min, andEA4=T - tA4=3.5 min – 2.50 min=1.00 min.• Extra time for possible pairs : (ER3, EA3), (ER4, EA4), (ER5, EA5) =(0.25, 0.667), (0.75, 1.0), (0.375, 1.25) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  29. Feasible Infeasible Infeasible Feasible (a) Possible NAi (b) Possible NRi EA4=1.00 Infeasible Feasible (c) Three possible pairs of (NRi, NAi) satisfying time and pairing constraints ER4=0.75 Analysis of Possible Action Strategies(2/5) ■ Graphical Representation of R and A’s Possible Action Strategies Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  30. Infeasible Feasible (c) Three possible pairs of (NRi, NAi) satisfying time and pairing constraints Analysis of Possible Action Strategies(3/5) • In terms of Gibsonian affordances, Fig. (c) provides the visualization of the domain to delineate openings and obstacles for R and A. •While (NR3, NA3), (NR4, NA4) and (NR5, NA5) are identified as three possible pairs of (NRi, NAi) to restore network communication within 3.5.min., extra times for those pairs would also vary from 0.25 to 0.75 min for R and from 0.667 to 1.25 for A. •If represented in a GUI appropriately, this information on R and A’s possible action strategies and their boundary conditions would greatly aid a mission supervisor’s decision making on which action R and A need to take in order to maximize their utilization or to increase the flexibility in his/her action planning. • One possible way to represent this information is to envision a time window (Rothrock 2001) representation of alternative action strategies. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  31. Analysis of Possible Action Strategies(4/5) ■ Time Window Representation (Rothrock, 2001) of R and A’s Possible Action Strategies and Their Bounds 0.75 1.00 Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  32. Analysis of Possible Action Strategies(5/5) • Two gray bars in Fig. (a) represent R and A’s travel times from their current locations to NR3 and NA3. •Two arrows attached to both ends of the bar represent R and A’s extra times that can be utilized for R and A’s other tasks before moving back to, on their ways to, or after arriving at NR3 and NA3. In other words, R can initiate its movement as early as at time 0 or as late as at time 0.25 in order to restore its communication with A within 3.5 min. Similarly, A can start its movement as early as at time 0 or as late as at 0.667. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  33. ■ A visualization framework based on Gibsonian-based has been proposed for representing a mobile robot and an astronaut’s possible strategies for maintaining network communication in a continuous and dynamic environment.■ The significance of our proposed work is not in the calculation of optimal solutions for travel, but rather in the graphical representation of bounds on actualized action strategies of the robot and the astronaut based on their possible action strategies. ■ On-going research to investigate the relationship between suitable circumstances and actualized potential actions. Summary Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  34. Investigating Compensatory and Noncompensatory Decision Making Strategies in Dynamic Task Environments Research Team Ling Rothrock, Yin Jing (Penn State) Alex Kirlik (UIUC) Sponsor: NSF DRMS Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  35. Decision Strategies • Compensatory: The weighting and summing of information using a linear-additive approach (i.e., linear regression). • Compensatory modes of decision strategies are contrasted against noncompensatory modes (Payne, 1976; Einhorn, 1970; Svenson, 1979) where, typically, not all available information is used and, therefore, trade-offs are often ignored. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  36. Decision Strategies (cont.) • Notable noncompensatory strategies: • Conjunctive – alternatives not meeting all thresholds are eliminated; • Disjunctive – accepts an alternative if one or more attributes are acceptable; • Lexicographic – prioritize the attributes and compares alternatives in descending order of attribute importance; • Elimination by Aspects – at each stage of the process, the dm selects an attribute and eliminates all alternatives that do not include a specified aspect until only one alternative remains; and • Take the Best – the decision maker uses a sequence of rules to choose an alternative in the face of uncertain knowledge. (Gigerenzer & Goldstein, 1996) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  37. Lunch Time Case 1. Consider an example of choosing a restaurant for an evening out. We assume that each prospective restaurant has been reviewed by a local newspaper food critic and that each has established a local reputation. We consider the following four hypothetical restaurants: RestaurantLocal ReputationNewspaper Review Desirability A 5.0 5.0 0.95 B 4.7 3.0 0.75 C 4.1 3.5 0.70 D 2.0 1.0 0.20 Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  38. Lunch Time (cont.) Case 2. Now consider the case where the decision maker is under time pressure so that he must choose a restaurant, eat quickly and return to a late business meeting. Therefore, while he is interested in eating well, he is more interested in finishing on time. Revisiting the four restaurants, we see the following: RestaurantSeating LineOccupancyDesirability A Full Full 0 B Full Seat Available 1 C Empty Full 1 D Empty Seat Available 0 Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  39. Lunch Time Models • Model for Case 1: • Model for Case 2: Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  40. Why Noncompensatory? • High task complexity and time stress (prevalent conditions in complex, dynamic supervisory control systems) trigger noncompensatory mode of decision making behavior (Payne, 1976; Payne et al., 1988; Wright, 1974). • Measurement scales • Regression analysis of criterion and predictors typically rely on interval or ratio scales. Can sometimes use ordinal scale and much more difficult with nominal scale. • Rules can naturally fit nominal up to ratio scales. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  41. Research Questions • How can we infer operator cue usage in complex and dynamic supervisory control environments? • How can we calculate lens model parameters in a noncompensatory modeling framework? Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  42. Research Question #1 How can we infer operator cue usage in complex and dynamic supervisory control environments? Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  43. (short) Introduction to Brunswik’s Lens Model • Judgment analysis is based on probabilistic functionalism • The chief task of psychology was to understand the functional relationship between a person and his environment • The essence of the person-environment relationship was necessarily based on uncertain relations among environmental variables • The environment in which a person is situated and the person himself should receive equal emphasis in psychological [decision] theory and research Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  44. The Lens Framework Proximal- peripheral Cues Distal variable Central response Achievement From Brunswik (1955) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  45. The Lens Model Equation – A Compensatory Representation Achievement (performance) is a function of the linear knowledge of the judge, the predictability of the task, the judge’s cognitive control, and unmodeled knowledge (Tucker, 1964). Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  46. The Lens Model Equation (cont.) • The multiple linear regression model of the judge is formulated as where • and e is the residual. The correlation between and • is given by and represents the cognitive control of the judge. • A corresponding multiple regression model, , is given for the ecology – graphically depicted as the left-hand side of the cues. In the case of the ecology model, represents the predictability of the criterion. Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  47. The Lens Model Equation (cont.) • The correlation between and , or G, has been labeled as linear knowledge (Hammond and Summers, 1972) to denote the linear correspondence between the judge’s decision policy and the optimal model of the criterion. • The correlation between the two sets of residuals ( and ), or C, is commonly called unmodeled knowledge – suggesting that if the residual variance is systematic, the judge is using a non-linear policy effectively. The remaining term, , is the cornerstone of the lens model and its equation is called the Lens Model Equation: Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  48. Problem Definition How do we incorporate noncompensatory strategies into the Lens Model framework? • Each candidate decision strategy is represented in disjunctive normal form. • Each candidate strategy is called a rule set (cf. the Pittsburgh approach, Smith, 1983). Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  49. Mathematical Development of Fitness Dimensions • Definition 1.1: An exemplar matrix, E, consisting of a set of binary variable vectors, called exemplars, whose range is the set {0,1}. Each exemplar within E is represented as ei, for i = 1, ..., m, where m is the total number of exemplars. Each binary variable within ei is represented as ei,j for j = 1, ..., n, where n is exemplarlength. Thus, E is a m by n matrix in the form: Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

  50. Fitness and Multi-objective Formulation (example - exemplars) Human Performance Assessment and Modeling Laboratory The Harold & Inge Marcus Department of Industrial and Manufacturing Engineering

More Related