“A GOMS Analysis of the Advanced Automated Cockpit”

1. “A GOMS Analysis of the Advanced Automated Cockpit” Conference Proceeding, CHI’94, “Human Factors in Computing Systems” (1994) By Sharon Irvong, Peter Polson,J.E. Irving - University of Colorado Institute of Cognitive Science Lecturers: Antoine JEGOU & Parry NG

2. Outline • Introduction • Why this study ? • Presentation of CDU/FMC • The GOMS model of FMC operation • Test description • Results • Conclusion / Discussion

3. Introduction • Similarities: • between the modern automated office and the high technologies flight deck • Both workers are managers of complex suites of automation Transfer knowledge • Goal of this article: Analyse the skills needed to perform tasks using the Flight Management Computer (FMC) on an advanced commercial aircraft. • Why ? Some reports on advanced aircrafts showed that… • High complexity of using FMC ; • Need months to have efficient skilled use of it. • Focus: …on using the interface to the FMC: the Control & Display Unit (CDU).

4. The Study • What Irving S., Polson P. and Irving J.E. have done ? • A detailed analysis of FMC tasks (using the GOMS model analysis) • Build a Training Program • Test the efficiency of this Training Program: • Trained subjects perform tasks in a full motion simulator • and compare them to others pilots

5. Control & Display Unit • The FMC is an automated device (Autopilot) • The CDU is one of the way to keep the pilot in the loop, for active monitoring • Before takeoff: • Initial position; • Route from origin to destination; • Other parameters. • In-Flight: • Changes given by the Air Traffic Control CDU-800

6. Design a training device… • Because it takes months to learn… • …researchers have designed a computer based training program, based on their GOMS analysis. • Test the efficiency of the training program • Results: •  Show that CDU is not optimal for ATC directives !

7. The GOMS analysis • GOMS=Goals, Operations, Methods & Selection Rules • Refer to to the work of Kieras D.E. and John B.E. for the GOMS model and later the EPIC model (based on CPM-GOMS), and Card S.K. in “The psychology of Human-Computer interaction” (1983) • Refer to presentations: • IEEM 552-7 : “Introduction to Cognitive Task Analysis”, Chipman, Schraagen, Shalin (1998) • IEEM 552-13 : “Predictive Engineering Models Based on the EPIC Architecture …”, Kieras, Wood, Meyer (1997) • Represents: “A formalism for representing the knowledge required to perform routine cognitive tasks”

8. The GOMS analysis • Goals = user’s intentions • e.g.: “I want to pilot an aircraft” • Can be divided into sub-goals: • I want to takeoff • I want to reach 10,000 feet • I want to land • Methods = the steps to achieve a goal • Operations= user’s actions • Selection Rules = choose between alternative methods

9. The INSTALL ROUTE call to method Access [route Area] Get FPF Company route id. Is cleared route a company route ? Install Manual Route Install Company route Designate [Company route id.] Designate [Departure Runaway] Insert [Company route id.] [Departure Runaway] Insert Verify [Route Entry] Verify Report Activate and execute NOW ? Activate & Execute The Pre-Flight task is a sequence of 6 calls to methods that accomplish the major sub-tasks involved, “INSTALL ROUTE” is one of them. Report

10. route Arrival JASIN JASON IOW departure The 3 Common Methods • ACCESS: • Identify the page associated with the current task • Carry out the sequence of operations necessary to get the page displayed on the CDU (I.e. function key or keyword) • DESIGNATE: • Determination of the identifier for a waypoint or an airway • INSERT: • Insert the information that was “designated” into the appropriate line

11. Examples… The Climb task: • A sub-task of the pre-flight task • Task requiring an ACCESS method • Is not provided by a prompt (not cued) • Use of function key… OAL • To respond to a directive from the ATC : “…Flight 123, You are cleared Coaldale” • ACCESS The LEGS page • Find “OAL”, DESIGNATE it • And INSERT it on line 1, left (“direct to”)

12. Evaluation of the model • Objective: • To implement a training program based on GOMS model of FMC tasks • To validate both the model and hypothesis that the skills represented in the model can be developed adequately and efficiently out of context of • Other cockpit automation • Device and displays • Full motion simulator

13. The Training Program • CDU emulation on a Macintosh platform • Highly constraints set of exercises • No precise description of the Training program

14. Transfer test Install route i.e. Flight route: Denver to Colorado Springs Pre-flight FMC directives came (e.g.:”” flight 123, you’re cleared Kiowa,…”) Direct To Going direct-to a point (not the original route) & add an waypoint clearance came (e.g.:”” hold at KIOWA,…”) Hold & Params Enter the holding parameter, back to holding fix via ATC clearance, “… cleared direct KIOWA at this time” Exit Hold Intercept Leg To Learns which runways are in use at destination Install Approach Intercept Leg To Intercept a point on instrument landing system

15. Subjects

16. Results (in general) • Difference in performance between groups for flight tasks were small • There are large differences between tasks ranging from 49% to 92% correct • Even the expert still made some errors !!! • Small differences between “200 pilots” and “Students” (experimental trained groups) in percent correct • both PRE-FLIGHT and MODIFY ROUTE (in-flight) tasks • “300 pilots” (professional trained groups) performed better than experimental trained groups especially on the pre-flight tasks • “Experts” did the best of all on in-flight tasks (not qualitatively better)

17. Results (Pre-flight tasks) For the average total times required to preflight the FMC, • highly significant differences between four groups, F(3,45) = 23.3, p<0.0001 For CLIMB task (i.e have a ACCESS problem), • Experimentally trained pilots  averaged 64 % correct • Professionally trained pilots  averaged 96 % correct

18. Results (Pre-flight tasks) • Reasons for the differences between experimental and professional trained groups: • experimental subjects: interacted for the very first time with actual aircraft hardware • “300 Pilots” had more experience with the actual hardware

19. Results (In-Flight tasks) • Highly significant difference between • groups (F(3,46) = 12.5, p <0.0001) and, • tasks (F(7,322) = 30.4, p < 0.0001) and, • group by task interaction (F(21,322) = 2.1, p < 0.003) • No consistent trend due to expertise across tasks

20. Experimental trained subjects Professional trained subjects Sample of In-flight task performance of all groups – mean time to complete and % correct

21. Results (In-Flight task – V83) • This task was the directive to add Victor 83, an airway, to the originally installed (programmed) route • Shows a large effect of expertise, • range from 25% correct for200 pilots to 90% correct for the Experts. • Little more than one-half the professionally trained pilot (“300”) could carry out this task without help

22. Results (In-Flight task -- V83) Comment on results: • identifier must be entered on the route (“RTE”) page • a large number of subjects tried to enter “V83” on the “LEGS” page getting the “NOT IN DATABASE” -- unclear system response • (i.e. ACCESS problem) • Although users may initially believe they got this message because they made a typographical error • no function key “AIRWAYS” to access the appropriate page to carry out the modification to the routing

23. Results (High-time task “INTERCEPT V83”) • Goal transformation : Re-mapping the clearance to CDU goal • “intercept V83”  “intercept the next down path waypoint on V83” • require the user to determine what the waypoint is (i.e. DESIGNATE problem) • “Expert’ had trouble completing this task without help

24. Results (High-time task “INTERCEPT V83”) Comment on results • Two set of target boxes shown on the CDU screen • left side: accomplishes a goal of “Direct To” • right side: accomplishes a goal of “Intercept-Leg-To” (Correct choice) • several subjects chose the left-hand side to place the waypoint identifier • (i.e. INSERT problem)

25. Results (Form-filling task “HOLD PARAMETERS” ) • involve checks between the current hold directive and the default value displayed on a page and some of those default values must be replaced (i.e DESIGNATE problem) • < 60 % Expert pilot carried out this task without prompting • Disconnection between the interface and the outer ATC environment • FMC does not operate in “radials” • Conversion between units are needed (125 radials  305 degrees)

26. Conclusion • The inconsistencies of the CDU (difficulties for all subjects) • “RTE” page not clear when looking for airways • Directives in radians, input in degrees • Low fidelity training !! • Some scores are far too low… (25% for “200 pilots” in the in-flight V83 task) • The GOMS model of CDU operation revealed • All tasks carried out by 3 common methods (ACCESS, DESIGNATE, INSERT) • But those methods are complex, heterogeneous, very memory-demanding • Future Work: A more precise analysis should reveal that • Tasks should be all cued (cf. CLIMB in the pre-flight task) • Supported by: The environment (ATC directives) & the CDU interface

27. Discussion • This paper does not go deep enough • But it is a conference paper which refers to a complete research paper ([7] in the reference) • Weakness of the paper: • Training: • Why this solution ? • How many hours of training ? • Test: • nber of trials ? • The use of videotaping is not described in details • The GOMS analysis seems to be good but on the other hand the training program very poor

28. Q/A…