Certification, Maintenance, and Operations: Strengthening the Process Links Dan Cheney, Carol Giles, and Ruth Harder Fed

1. Certification, Maintenance, and Operations: Strengthening the Process LinksDan Cheney, Carol Giles, andRuth HarderFederal Aviation Administration

2. 19th Annual FAA/JAA International Conference Phoenix, Arizona June 5, 2002

3. Workshop Item III Review of CPS Study • CPS Overview • Key Findings and Observations • Conclusions

4. CPS Overview • The remarkable safety of today’s commercial aviation system is the product of continuously identifying and applying improvements to all aspects of safety by manufacturers, operators, and governments • Continuous improvements involving: • Requirements • Processes • Technology • Training

5. CPS Overview • During the course of investigation of the January 2000, Alaska Airlines Flight 261 MD-83 accident, particular attention has been given to the process associated with how critical airplane systems are certified and eventually maintained and operated • Although the NTSB investigation of the Alaska 261 accident is not yet complete, in December 2000, the FAA initiated a review of major processes being used in the U.S. to certify commercial transport airplanes

6. CPS Overview • The Commercial Airplane Certification Process Study (CPS) began its work in January 2001, and has focused on key aspects of the airplane certification process and the interrelationship of these processes to certain operation and maintenance processes • 34 team members from government, industry, academia, and consulting organizations

7. OVERSIGHT BOARD Mac Armstrong Bob Davis Nick Sabatini John Goglia Lou Mancini Daniel Schrage Chairman STUDY TEAM Dan CheneyChairman Frank FickeisenCo-Chairman Julie ZacharyAdmin Support Academia (3) Airlines (8) ALPA (2) DoD (2) FAA (10) FPA (1) INTL (1) MANUF (4) NASA (2) CPS Overview

8. CPS Overview • The CPS review represents a critical self-examination of current processes in order to identify areas for improvement • In order to identify improvements in an already very safe system, this kind of introspection is considered necessary • CPS was not an accident investigation activity; accidents and incidents were reviewed only to provide insight into process issues

9. CPS Overview • CPS review timetable of one year to study and report on key aircraft certification, maintenance, and operational processes • Tasked to: • Provide a report to senior FAA management with findings • Identify areas for process improvements

10. Certification Operations and Maintenance CPS Overview • The CPS report was completed in early March 2002, and submitted to the Associate Administrator for Regulation and Certification, Mr. Nick Sabatini • The study paid particular attention to the adequacy of the processes related to the “arrows” in this figure

11. CPS Overview • The CPS team met formally 15 times, averaging approximately one week a month • Some groups met between formal meetings • A very large amount of information was analyzed during the study • 68 case studies (accidents/incidents) • 42 presentations • 12 interviews • 10 historical reports

12. CPS Overview

13. CPS Overview • Five focus areas were identified through multiple screening and selection processes • Safety Assurance Processes • Aviation Safety Data Management • Maintenance/Operations/Certification Interface • Major Repairs and Modifications • Safety Oversight Processes • Findings and observations resulted from an in-depth review in these five areas

14. Key Findings and Observations • There are 15 findings and 2 observations in the CPS report • These represent areas where the CPS team felt that existing processes could be improved. • Although close examination of any of the 15 findings could lead one to believe the current system has significant deficiencies, it should be noted that the study team deliberately looked for problem areas so as to improve the system

15. Key Findings and Observations • Rather than go into detail on all 15 findings, I would like to mention certain key findings, some from each of the five focus areas • Most findings have strong international ties • An in-depth discussion of all 15 findings and 2 observations can be found in the final report

16. Finding 1 • Human performance is still the dominant factor in accidents: • The process used to determine and validate human responses to failures and methods to include human responses in safety assessments need to be improved. • Design techniques, safety assessments, and regulations do not adequately address the subject of human error in design or in operations or maintenance. • 80% of all accidents have a human error contribution • 66% of all accidents identify flight crew errors as primary cause • Rates unchanged despite technological advances

17. Finding 1 • Human errors involve many different factors and issues - difficult to identify • Represents major opportunity for safety improvements • Better understanding of range of human skills needed • Lessons learned knowledge needed by all those with human factors responsibility • May benefit from human engineering “best practices” sharing within industry • FAA Human Factors Team report currently addressing many of these issues

18. Finding 4 Processes for identification of safety critical features of the airplane do not ensure that future alterations, maintenance, repairs, or changes to operational procedures can be made with the cognizance of those safety features. • Many critical safety features of complex transport airplane designs are not readily obvious. Examples: • Check valves, shear links • Environmental capability features (e.g., lightning) • Seals, drain lines, vapor barriers • Wire routing, electrical grounding paths • Secondary structural load paths, energy absorption devices

19. Finding 4 • Increased awareness of safety critical features of the airplane is needed by those operating and modifying it • Strongly linked to several other findings in study, e.g., Finding 7, which is discussed next • Represents a high-leverage opportunity for improvement • EAPAS and recent FAR 25 (i.e., fuel system safety) change cited as good examples for raising awareness

20. Finding 7 There is no widely accepted process for analyzing service data or events to identify potential accident precursors. • All safety data systems should have common objective - to identify the need for safety intervention prior to an accident • Some government/industry safety data management programs have shown promise in accident precursor capability • May represent “best practices” approach for the rest of industry • CAAM, COSP, FOQA and ASAP are examples

21. Finding 8 Adequate processes do not exist within the FAA or in most segments of the commercial aviation industry to ensure that the lessons learned from specific experience in airplane design, manufacturing, maintenance, and flight operations are captured permanently and made readily available to the aviation industry. The failure to capture and disseminate lessons learned has allowed airplane accidents to occur for causes similar to those of past accidents. • “Lessons learned” information needs to be an important part of accident precursor recognition/intervention strategy • Currently, no formal “lessons learned” database exists

22. Finding 8 • Most safety “lessons learned” are still relevant today despite technology advances • “Corporate memory” fades with time and loss of experienced personnel • Some informal OJT attempts to pass critical safety lessons from one generation to the next, but becoming increasingly difficult • Accident investigation findings often narrow focused while important high level lessons usually have broad applicability

23. Finding 10 There are currently no industry processes or guidance materials available that ensure: • Safety related maintenance or operational recommendations developed by the OEM are evaluated by the operator for incorporation into their maintenance or operational programs. • Safety related maintenance or operational procedures developed or modified by the operator are coordinated with the OEM to ensure that they do not compromise the type design safety standard of the airplane and its systems. • Challenge is for “safety related” or “significant” material to be coordinated rather than all material • Not all safety related information can/should be the subject of airworthiness directives

24. Finding 10 • This is not an easy subject to solve and may benefit from industry partnership, both manufacturers and operators • Method for ensuring distribution and review of safety material is needed rather than resorting to airworthiness directives (ADs) • If quickly implemented, the solution to this finding could have an immediate impact on safety

25. Finding 13 Inconsistencies exist between the safety assessments conducted for the initial Type Certification (TC) of an airplane and some of those conducted for subsequent alterations to the airplane or systems. Improved FAA and industry oversight of repair and alteration activity is needed to ensure that safety has not been compromised by subsequent repairs and alterations. • Vast majority of repairs/alterations are done consistently with the safety assessment of the original design • However, cases exist of inadequate repairs and alterations • FAA’s “STC Process Review” and “Field Approval Process” are examples of efforts to improve this

26. Finding 15 Processes to detect and correct errors made by individuals in the design, certification, installation, repair, alteration, and operation of transport airplanes are inconsistent allowing unacceptable errors in critical airworthiness areas. • Transport airplane safety achieved, in large part, through fail-safe, redundant, and fault-tolerant design concepts • These concepts are not extended consistently to single-point human error potential during maintenance and alteration • Strongly linked to Finding 4 (Safety Critical Features)

27. Conclusions • Four common areas were identified, which appeared to link the findings and observations: • Information Flow: Barriers to critical information flow may exist • Human Factors: Failure of the human machine interface • Lessons Learned: Significant safety issues learned through accident/incident analysis • Accident Precursors: Significant incidents that are indicators of a serious service problem requiring intervention in order to prevent an accident

28. Conclusions

29. Certification AREAS INTERFACE Operations and Maintenance Conclusions • CPS findings and observations, as well as many of the accidents analyzed during this study, concern interface issues involving certification processes and operation/maintenance processes

30. Conclusions • “Cross discipline” consideration will be essential in order to achieve airplane level safety awarenessduring safety decision making

31. Conclusions • Improvements in safety will require coordinated initiatives involving the manufacturer, operator, and FAA • Accident precursor recognition and intervention will require greater airplane level safety awareness during all aspects of the commercial airplane life cycle

32. Workshop Item IV CPS Response Team Strategic Plan • Overview of CPS Change Areas • CPS Change Areas & International Links • Potential for Future Direction

33. Change Area Overview • CPS Response Team identified four Change Areas from CPS Report: I. Safety Information Awareness II. Human Factors Integration III. Repairs & Alterations IV. AIR/AFS Integration • The Change Areas provide a framework for CPS implementation in the Strategic Plan

34. CPS Change Area Overview • The Change Areas are separate initiatives for safety improvements, but have strong interrelationships with each other. • Work on Change Areas can be initiated in parallel. • Breaking Change Areas into smaller packages would not provide necessary integration during implementation

35. I. Safety Information Awareness • Issue: Safety information processes used throughout industry are fragmented and uncoordinated, causing inefficient identification and mitigation of safety issues • Key Topics to be addressed in this Change Area: A. Safety Critical Design Information (F 2, 3, 4) B. COS Data Management (F 5, 6, 7, 9) C. Lessons Learned/Airplane System Awareness (F 8) D. OEM/Operator Safety Info Transfer (F10, O1)

36. I. Safety Information Awareness • To raise awareness of safety issues and mitigate them, safety information must be managed in an integrated way: • Identify important safety critical features in the design • Define how to communicate this information • Timely identify safety issues from in-service data and mitigate the safety concern • Capture and retain Lessons Learned

37. I. Safety Information Awareness • The Response Team believes this Change Area provides the strongest opportunity for improvements to safety • Safety information should be addressed at the global level to be most effective; otherwise key information may be missed • Plan for Change: Breakout teams will developed detailed work plans for each Information Topic.

38. II. Human Factors Integration • Issue: Human errors continue to dominate as a contributing factor in accidents (F1,15) • Multiple initiatives underway today internationally to address human factors issues, are fragmented & uncoordinated • Plan for Change: Urgent need for a comprehensive, joint industry/government plan that integrates these separate efforts.

39. II. Human Factors Integration • HF Integrated plan should: • Focus on human factors issues that have resulted in accidents in the past and/or that could result in accidents in the future. • Bring together the various certification, operations, and maintenance efforts currently underway within the FAA and industry. • Address human factors inputs into the ongoing operational safety decision process for the existing fleet.

40. III. Repairs & Alterations • Issue: Lack of clear requirements for major repairs and alterations, combined with inadequate oversight (F12, 13, 14, O2) • Changes made without understanding of the original certification assumptions add risk because the modifier, maintainer, or operator may not understand safety implications. • Global impact due to the number and complexity of repairs and alterations being accomplished worldwide on a daily basis

41. III. Repairs & Alterations • Plan for Change: • Review industry best practices to develop a model and issue guidance material for establishing repair and alteration decision logic. • Provide guidance to insure alterations consider original OEM safety critical information • Review existing policies and procedures related to the oversight of consultant DERs who approve complex repairs, alterations, or STCs

42. IV. AIR/AFS Integration • Issue: Lack of effective communication and coordination between FAA offices responsible for certification (AIR) and maintenance and operations (AFS) may compromise safety (F11) • Affects the FAA’s ability to address industry safety issues effectively and industry’s ability to comply fully

43. IV. AIR/AFS Integration • Plan for Change: • Define AVR-level policy directed towards improved internal and external communication and coordination between AIR and AFS. • Develop business processes to enhance coordination between AIR and AFS. • Improved FAA in-house communication between these organizations will achieve safety benefits from more timely and consistent communication with global counterparts and industry

44. CPS Future Direction • Review of CPS planned safety improvements in relation to other major safety initiatives - identify and manage the relationships between change initiatives • CPS Strategic Plan Phase II work will include detailed plan development, using breakout teams comprised of people from all involved FAA and industry organizations

45. CPS Future Direction • All parties (manufacturers, operators, and authorities) must be fully involved in and committed to the CPS safety solutions. • Change will be necessary in many or all involved organizations, not just within the FAA or any single organization. • Airplane System Awareness must continue to improve throughout FAA and industry worldwide.