1 / 49

The Apex Architecture Multitasking Under Time-Pressure and Uncertainty

The Apex Architecture Multitasking Under Time-Pressure and Uncertainty. Michael Freed NASA Ames Research Center. Cognitive Architecture Applications at NASA. Evaluating cockpit designs Planning crewed missions Artificial participants in training/eval simulations

lilith
Download Presentation

The Apex Architecture Multitasking Under Time-Pressure and Uncertainty

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Apex ArchitectureMultitasking Under Time-Pressure and Uncertainty Michael Freed NASA Ames Research Center

  2. Cognitive Architecture Applications at NASA • Evaluating cockpit designs • Planning crewed missions • Artificial participants in • training/eval simulations • Tech impact assessment • Problem: • Demand for human simulation • models exceeds what modelers • can supply with current tools

  3. Cognitive Architectures as Practical Tools • What capabilities/resources does it provide above • those in a general-purpose programming language? • How usable are those capabilities? How much time, • expertise, and inventiveness does someone need to • build an agent that meets given requirements? • How good are its software qualities? E.g. stability, • maintainability, extensibility (“ilities”)

  4. Outline • Overview of Apex architecture and applications • Capabilities incorporated into the architecture • - Procedure-based reactive planner • - Multitask management • Usability-related efforts

  5. Apex applications Realistic simulations of human pilots and air traffic controllers Interface evaluation based on CPM-GOMS Autonomous robots for reconnaisance and surveillance

  6. CapabilitiesWhat we need Many domains of practical interest are demanding in the sense that a skilled agent must : • Cope with uncertainty • Can’t completely know or predict world state • Actions may fail or produce undesirable side-effects • Actual resource requirements determined during execution • New, urgent tasks can arise at any time • Cope with time-pressure • Can’t deliberate endlessly • Options for resolving uncertainty constrained • Conform to SOPs (not just goal-pursuit)

  7. CapabilitiesProcedure-based reactive planner • Cope with time pressure by… • Using stored, generalized procedures • Avoiding expensive decision processes (search) • Cope with uncertainty by… • Deciding action at the last moment when maximum situation information available • Integrated contingency handling • Not necessarily sound or complete Procedure-based reactive planners (E.g. RAPS, ESL, PRS) select action based on stored, generalized procedures.

  8. Procedure representation in Apex • concurrency (procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))

  9. Procedure representation in Apex • concurrency • reactivity / cl-control (procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))

  10. Procedure representation in Apex • concurrency • reactivity / cl-control • hierarchy / selection (procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))

  11. Procedure representation in Apex • concurrency • reactivity / cl-control • hierarchy / selection • contingency-handling (procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))

  12. Procedure representation in Apex • concurrency • reactivity / cl-control • hierarchy / selection • contingency-handling • multitask management (procedure (index (hold-altitude using mcp)) (profile right-hand) (step s1 (clear right-hand)) (step s2 (find-loc alt-hold-button => ?loc)) (step s3 (press-button ?loc right-hand) (waitfor (empty right-hand) (location alt-hold-button ?loc))) (step end (terminate) (waitfor (illuminated alt-hold-button)) (step aux1 (restart ?self) (waitfor (resumed ?self))))

  13. Execution-time action-selection using stored procedures • Execution-time action selection • + Copes well with uncertainty • Reduced deliberation • Stored procedures • + Natural representation of SOPs • + Easy to construct, debug, evaluate • + Fast decisions • + Can adapt to statistical structure of env by repeated use • + Can adapt environment to make procedures more effective • + Easy to communicate to other agents • - Not necessarily sound or complete • Successful procedure-based behavior requires multitasking

  14. Multitask management examples • Delay answering phone until finished typing sentence heuristic: prefer to delay interrupt until good stopping point • Pull over to side of road before studying map • Drive back onto road (but don’t drive to start point) • Do something useful when stopped at a red light • … Multitask management: execution-time coordination of interacting tasks based on task-specific characteristics and general heuristics.

  15. Challenge Support creation of agents with human-level ability to employ diverse multitask management tactics • General heuristics underlying tactics • architecture mechanisms • Task-specific knowledge • specialized representation elements • task representation methodology

  16. Multitasking in Apex • Concurrency control • Interruption, resumption, interleaving • Efficient use of resources

  17. Concurrency Control PDL idioms Race (procedure (index (do-it)) (step s1 (do-A) (step s2 (do-B) (step s3 (do-C) (waitfor ?s1) (waitfor ?s2)) (step s4 (terminate) (waitfor ?s3))) Synchronize (procedure (index (do-it)) (step s1 (do-A)) (step s2 (do-B) (waitfor (started ?s1))) (step s3 (terminate) (waitfor ?s1 ?s2))) Converge (procedure (index (do-it)) (step s1 (do-A) (step s2 (do-B) (step s3 (do-C) (waitfor ?s1 ?s2) (step s4 (terminate) (waitfor ?s3)))

  18. Interruption and resumption Determining if tasks conflict • Resource requirements declared in procedure (profile (<resource> [tolerance]) …) (hold-resource <resource>) [tolerance]) (release-resource <resource> [tolerance]) • Some tasks tolerate brief interruptions • Conflict exists between tasks A and B if • A and B both require resource R, and • Expected Duration (A) > Tolerance (B) or Expected Duration (B) > Tolerance (A)

  19. Interruption and resumption Resolving task conflicts • Compute priority from task attributes & situational factors: • urgency (U): measure of time until deadline • importance (I): expected cost of missing deadline • subjective workload (S): measure of task crowding • interrupt cost (IC): expected cost of interruption • In low workload, prefer urgent tasks (avoid unnecessary loss) • In high workload, prefer important tasks (avoid most expensive losses)

  20. Interruption and resumption Resolving task conflicts Resource conflicts resolved using priority heuristic Priority information set with step-level PDL clauses (step … (priority <urgency> <importance>) (interrupt-cost <cost>)

  21. Interruption and resumptionTransition behaviors (procedure (index (fly-cruise-leg using manual-control)) (step s1 (maintain-altitude) (interrupt-cost 5)) ... (step s12 (handoff-to-pilot-not-flying) (priority (importance 10) (urgency 10))) (waitfor (interrupted ?self))) (step s13 (monitor-pilot-not-flying) (waitfor (completed ?s12))) (step s14 (request-role-pilot-flying) (waitfor (resumed ?self))) ...)

  22. Efficient use of resources • Combine redundant tasks (merge <condition> [<task pattern>]) • Exploit slack in procedure resource use • Automatic detection of slack intervals • Integrated dispatch scheduling • Concurrent recursive decomposition => tasks • Priority-based allocation => schedule

  23. Multitask Management Summary • Multitasking ability founded on tactical knowledge derived from general heuristics • Reactive planners can be extended to execute these heuristics in uncertain/time-pressured environments • Specifically, extensions for concurrency control, interruption handling and resource management facilitate use of multitasking tactics • Understanding of what needs to be represented and what notation is best for this purpose are improving as new Apex applications are developed

  24. Motivation and Learning Work in progress • Intent inference for multitasking agents • use procedural knowledge in reverse • distinguish separate “threads” of behavior • initial approach in Freed and Dahlman, 2002 • Learning aggregate properties of procedures • learning duration and resource characteristics • will enable better dispatch scheduling • other statistics useful for handling contingencies • such as failure and interruption

  25. Reducing time/expertise to construct models • Usability efforts • High-level language

  26. Behavior representation: usability Intuitive (step s3 (stop) (waitfor (shape ?x light) (color ?x red))) conjunctive preconditions mutually constraining Expressive (step s4 (slow) (waitfor (color ?x green) then (color ?x yellow)) able to express temporal relations between preconditions Compact abbreviated form for sequential procedures Evolves as users’ needs become better understood Support tools exist for authoring and debugging

  27. Reducing time/expertise to construct models • Usability efforts • High-level language • Visualization/debug tools

  28. Visualizing and Debugging • Sherpa • Simulation traces • Physical environment • PERT charts • VISTA • Enhanced Sim Trace • (2001 CMU student project)

  29. Reducing time/expertise to construct models • Usability efforts • High-level language • Visualization/debug tools • World modeling

  30. Time-consuming and error-prone when done by hand Specifying a Physical World Model

  31. Specifying a Physical World Model Easy with drag-and-drop interface CMU 2002 student project

  32. Reducing time/expertise to construct models • Usability efforts • High-level language • Visualization/debug tools • World modeling • Interoperability

  33. Interoperability Apex interops with: Riptide X-Plane DOMS Mozilla AMBR (HLA) Reusable elements Apex API Sim compatibility Com support Apex pilot flies an F-16 over NASA Ames

  34. Reducing time/expertise to construct models • Usability efforts • High-level language • Visualization/debug tools • World modeling • Interoperability • Reusable components

  35. 0 new-cursor-location 290 perceive- target 50 50 50 50 START initiate-move- cursor verify- target pos attend- target initiate- POG initiate- mouseDn 590 (FL) 100 100 mouseDn mouse Up move- cursor 30 from Gray and Boehm-Davis (2000) POG Resuable building-blocks (procedure (index (fast-move-click-R-hand-on-mouse :target ?target)) (step c1 (initiate-move-cursor ?target)) (step m1 (move-cursor ?target) (waitfor ?c1)) (step c2 (attend-target ?target)) (step c3 (initiate-eye-movement ?target) (waitfor ?c2)) (step m2 (eye-movement ?target) (waitfor ?c3)) (step p1 (perceive-target-complex ?target)) (step c4 (verify-target-position ?target) (waitfor ?c3 ?p1)) (step c5 (initiate-click ?target) (waitfor ?c4 ?m1)) (step m3 (mouse-down ?target) (waitfor ?m1 ?c5)) (step m4 (mouse-up ?target) (waitfor ?m3)) (step t (terminate) (waitfor ?m4 ?rvr1 ?rvr2)))

  36. Reducing time/expertise to construct models • Usability efforts • High-level language • Visualization/debug tools • World modeling • Interoperability • Reusable components • Distributed development

  37. Distributed Development • Why needed • Library of “building blocks” • Resource models • Software interfaces • Visualization tools • Making it happen • Software infrastructure • Educational outreach • Web-based support Apex collaborative web site (CMU HCI project 2001)

  38. Apex Modeling Tool • Apex System • Apex architecture (autonomy) • Human resource architecture • Reusable procedure lib • World model widget libs • Simulation “engine” • GUI-based vis/debug tools • GUI-based world dev tool • Interoperability framework • Manuals, tutorials, … Download system from:http://human-factors.arc.nasa.gov/apex/ User Support: apex-help@eos.arc.nasa.gov

  39. Apex 2.3 Apex is available at human-factors.arc.nasa.gov/apex

  40. How PDL specifies interleaving • CPM-GOMS requires that actions from templates earlier • in a template sequence have priority in resource conflicts • over activities from later templates. • Initial approach used the priority clause: • determines how to resolve resource conflict • globally scoped to allow comparison of any tasks • (step s2 (turn off alarm) • (waitfor (on alarm)) • (priority 5))

  41. How PDL specifies interleaving Global scope of priority value assignments undesirable for CPM-GOMS representations (procedure (procedure (index (get ?amt from atm)) (index (init atm transaction)) (step s1 (init atm transaction) (step s1 (insert card) (priority 3000)) (priority 3200)) (step s2 (withdraw ?amt)) (step s2 (enter password) (priority 2000)) (priority 3100)) (step s3 (end atm transaction) (step s3 (terminate) (priority 1000)) (waitfor ?s2))) (step s4 (terminate) (waitfor ?s3))) Requires modeler to anticipate decomposition Complex procedure to set priority values

  42. How PDL specifies interleaving Solution alternative mechanisms/syntax for resolving resource conflicts that is dynamically scoped (procedure (procedure (index (get ?amt from atm)) (index (init atm transaction)) (step s1 (init atm transaction) (step s1 (insert card) (rank 1))(rank 1)) (step s2 (withdraw ?amt)) (step s2 (enter password) (rank 2))(rank 2)) (step s3 (end atm transaction) (step s3 (terminate) (rank 3)) (waitfor ?s2))) (step s4 (terminate) (waitfor ?s3)))

  43. How PDL specifies interleaving Abbreviation to further simplify syntax… (procedure :ranked (procedure :ranked (index (get ?amt from atm)) (index (init atm transaction)) (step s1 (init atm transaction) (step s1 (insert card) (step s2 (withdraw ?amt)) (step s2 (enter password) (step s3 (end atm transaction) (step s3 (terminate) (step s4 (terminate) (waitfor ?s3))) (waitfor ?s2)))

  44. 0 new-cursor-location 50 50 verify- target pos initiate- mouseDn 695 545 (FL) 100 move- cursor mouseDn 100 30 POG initiate- POG perceive- target attend- target 50 50 START initiate-move- cursor END FAST MOVE-CLICK - from Gray and Boehm-Davis (2000)

  45. Interleaving Templates 0 0 new-cursor-location new-cursor-location 290 290 perceive- target perceive- target 50 50 50 50 50 50 50 START initiate-move- cursor attend- target initiate- POG verify- target pos 50 START initiate-move- cursor attend- target initiate- POG verify- target pos initiate- mouseDn initiate- mouseDn 590 (FL) 182 (FL) 100 100 100 100 mouseDn mouse Up move- cursor mouseDn mouse Up move- cursor FAST M/C FAST M/C 30 30 POG POG last vision action in the previous operator

  46. Autonomous Helicopter ApplicationProject Lead: Matt Whalley (Army)Autonomy Lead: Michael Freed (NASA/IHMC) Intelligent recon/surveillance Goal – maximize information return Task – decide where to go next Ariel (Yamaha RMAX) Payload: 66 lbs. Flight duration: 60 min

  47. Fire Theft cost cost time time Recon/surveillance target selection • Utility of target selection • Set of target sites: s є S • Observation time-cost • distance • flight dynamics • required observe action • Criterial events: ei,s є Es • baseline p(ei,s ) and mods • cumulative cost(ei,s ,t) • Approach – multimethod • search for small target sets • for larger sets, method depends • on uniformity/clumpiness • spatial clumps • cost and “deadline” clumps • Research Area – resource mgt. • Target selection is special case • of allocating unary resources • under time-pressure/uncertainty. Best choice = min opportunity cost

  48. bootstrap enable select match refine allocate execute start terminate Apex Architecture Execution-time action-selection using stored-plans Exec-time Action-Selection + High uncertainty tolerance - Little time to deliberate Stored Plans + Fast decisions - Not sound or complete + Adapted to stat struc of env + Mutual evolution with env + SOP compliance + Easy to construct, debug + Amenable to expert evaluat’n + More predictable behavior

More Related