FAA Oversight of Safety-Critical Software Systems: A Case Study

1. FAA Oversight of Safety-Critical Software Systems: A Case Study William S. Greenwell Department of Computer Science University of Virginia

2. August 6, 1997 1:42 AM Korean Air flight 801 impacts terrain on final approach to Guam Int�l runway 06L. En-route from Kimpo Int�l, Seoul, Korea. Boeing 747-300 carrying 254 people 228 Fatalities, 26 Seriously Injured Instrument meteorological conditions prevailed at time of accident.

3. Runway 06L ILS Approach NOTAM: Runway 06L glideslope unusable Without glideslope, pilots must execute nonprecision approach using step-down fixes. Runway threshold D3.3 from UNZ VOR.

5. Wreckage of KA Flight 801

6. Proximity to UNZ VOR

7. NTSB Findings Captain lost awareness of aircraft�s position on the landing approach. (p. 173) Captain possibly believed UNZ VOR was co-located with the runway, causing him to descend below intermediate approach altitudes. (p. 173) Classified as Controlled Flight Into Terrain.

8. Barriers to CFIT On-board Barriers: Instrument Landing System (ILS) Localizer/Glideslope Indicators Outer/Middle/Inner Markers Distance Measuring Equipment (DME) Published approach procedures Ground Proximity Warning System (GPWS) Pilot not flying, flight engineer Ground-based Barriers: Minimum Safe Altitude Warning (MSAW) system

9. On-Board Barriers Instrument Landing System (ILS) Glideslope out of service since July Outer/Middle marker indicators suppressed Published approach procedures Pilot misunderstood/disregarded DME fixes. Ground Proximity Warning System (GPWS) Commonly ignored due to nuisance warnings Pilot not flying, flight engineer Didn�t challenge approach soon enough

10. Ground-based Barriers Minimum Safe Altitude Warning System Inhibited by FAA due to nuisance warnings NTSB concluded: �Contributing to the accident was the Federal Aviation Administration�s (FAA) intentional inhibition of the minimum safe altitude warning system (MSAW) at Guam and the agency�s failure to adequately manage the system.� (p. 175)

11. MSAW Overview Developed by FAA in response to NTSB Safety Recommendation A-73-46. Incorporated into ARTS IIA in 1990. Data Inputs: ARTS � track positions & altitudes Terrain database � elevation data Configuration file � airport & runway information, service area definitions General & approach path monitoring The intention of the MSAW system was to alert controllers to low-flying aircraft so that they could in turn notify the flight crew, in the event that the flight crew was unaware they were below altitude.The intention of the MSAW system was to alert controllers to low-flying aircraft so that they could in turn notify the flight crew, in the event that the flight crew was unaware they were below altitude.

12. MSAW General Monitoring

13. MSAW Approach Path Monitoring

14. MSAW Deployment Installed at 193 ARTS IIA & ARTS III sites. Each site employed customized terrain database and configuration file. Individual sites free to inhibit processing as needed to alleviate nuisance warnings. No guidance for defining inhibit zones No oversight of site adaptations Waiver required to turn off MSAW entirely.

15. Guam MSAW Chronology 1990 � MSAW incorporated into ARTS IIA Originally configured with 55-nm service area. March 1993 � Guam adapts MSAW parameters to include 54-nm inhibit zone. February 1995 � New MSAW build becomes operational with inhibit zone. July 1995 � Facility evaluation of Guam notes inhibition as �informational� item.

16. Guam Inhibit Zone

17. Guam Chronology / Cont. February 1996 � NOAA delivers new terrain database for Guam MSAW system. April 1996 � New MSAW build becomes operational with updated terrain database & 54-nm inhibit zone. May 1997 � FAA reevaluates Guam ATC facility, but does not note MSAW inhibition. August 6, 1997 � KA flight 801 accident

18. Effectiveness of MSAW NTSB/FAA simulation indicated that, without the inhibition, MSAW would have generated an alert 64 seconds before impact. NTSB: This would have been sufficient for the controller to advise KA 801. (p. 174) NTSB concluded that FAA�s quality assurance of MSAW was inadequate.

20. FAA�s Post-accident Actions Recertified MSAW at all 193 equipped sites. Two other improperly configured systems Instituted policy for periodic recertification. MSAW inspection added to facility evaluation. Developed standards for site adaptation. Centralized configuration management. All configuration changes made by AOS. AOS used to stand for �Airway Operational Support�, which has since been renamed the �Operational Support Directorate�, but still uses the AOS label.AOS used to stand for �Airway Operational Support�, which has since been renamed the �Operational Support Directorate�, but still uses the AOS label.

21. Underlying Cause FAA changed a safety-critical system without reexamining the scenario the system addressed. Trial-and-error approach to site adaptation Sites permitted to make changes at their discretion without review or recertification. No instructions or guidance for making changes FAA allowed Guam to operate normally despite having two safety-critical systems out of service.

22. MSAW: Safety-Critical? FAA: Safe operation of aircraft is ultimately pilots� responsibility. No certification criteria for ground systems MSAW merely an aid to AT controllers But� MSAW only ground-based CFIT barrier FAA NFSD manager: MSAW �safety-critical item� [1] FAA. �FACT SHEET: FAA Actions on Aviation Safety Relating to KA 801 Accident.� 28 March 1998. [2] NTSB. Public Hearing in Correction With the Investigation of Aircraft Accident, Korean Air , Flight 801,B-747-300, Agana, Guam, August 6, 1997. 24 March 1998. [1] FAA. �FACT SHEET: FAA Actions on Aviation Safety Relating to KA 801 Accident.� 28 March 1998. [2] NTSB. Public Hearing in Correction With the Investigation of Aircraft Accident, Korean Air , Flight 801,B-747-300, Agana, Guam, August 6, 1997. 24 March 1998.

23. Lessons for the FAA Systems that serve as barriers�even ground systems�provide safety-critical functions to larger systems. When changing such a system, we must examine how that change will affect the safety of the overall system. How do we reduce nuisance warnings without compromising alerting capability?