Practical overview of the LOSA process - PowerPoint PPT Presentation

isaac
line operations safety audit losa a practical overview n.
Skip this Video
Loading SlideShow in 5 Seconds..
Practical overview of the LOSA process PowerPoint Presentation
Download Presentation
Practical overview of the LOSA process

play fullscreen
1 / 38
Download Presentation
Practical overview of the LOSA process
1433 Views
Download Presentation

Practical overview of the LOSA process

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. The University of Texas Human Factors Research Project Third ICAO-IATA LOSA & TEM Conference Malaysia Airlines, Kuala Lumpur - September 13-14, 2005 Line Operations Safety Audit (LOSA): A Practical Overview James Klinect and Patrick Murray The University of Texas / The LOSA Collaborative

  2. Introduction: James Klinect • University of Texas (UT) – LOSA research for the industry • Project Coordinator – TEM & LOSA • Director: Bob Helmreich • LOSA Collaborative – LOSA implementation for airlines • Role: Founder/President • De-identified data to University of Texas • Disclaimer: Not an commercial for the LOSA Collaborative • Sharing of mistakes made and lessons learned • Provide a enough information to evaluate LOSA and possibly implement it themselves (ex, Varig, Lan Chile,….)

  3. AeroMexico Asiana Airlines Alaska Airlines Air New Zealand Braathens Cathay Pacific China Airlines COPA LOSA Airlines: 1996-2005 Since 1996, 4500+ observations, 25 airlines & 11 countries Continental Airlines Continental Express Continental Micronesia Delta Airlines EVA Air Frontier Airlines LACSA QANTAS Malaysia Airlines Mt. Cook Regional Express Singapore Airlines Silk Air TACA / TACA Peru UNI Air US Airways

  4. Presentation Outline • LOSA Rationale / Deliverables • Why LOSA and what do you get? • LOSA Operating Characteristics • What is the definition of LOSA and how does it work? • LOSA E-mail Inbox • What are the most frequently asked LOSA questions? • Audience Question and Answer

  5. LOSA Rationale / Deliverables

  6. Safety Aviation Safety Envelope Accidents Incidents Normal Operations

  7. Voluntary Incident Reports LOSA QAR Mandatory Incident Reports Accident Investigation Safety Data Coverage Accidents Incidents Normal Operations

  8. Aviation Safety Envelope Accidents Incidents Normal Operations LOSA Rationale Proactive snapshot of system / crew performance strengthsand weaknesses in normal operations (just like a health checkup)

  9. LOSA Measurement Challenge • Challenge - What type of observational measure best captures safety performance data in normal operations? • Concerns • Observer overload – Can’t capture everything • Data analyst overload – Must have a roadmap for interpretation • Manager overload – Data need face value and point to safety issues • Solution – Threat and Error Management (TEM)

  10. Error Management 1 Manage operational complexity Threat Management LOSA Measures 2 Manage their own errors 3 Manage aircraft deviations Undesired Aircraft State Management Primary LOSA Measures: TEM What do flight crews do to safely fly from A to B? Where’s CRM and human factors? Non-technical skills are one of many contributors to effective/ineffective TEM performance

  11. Most frequent and mismanaged threats Most frequent and mismanaged errors Fleet and operational differences Comparison to other airlines on LOSA safety indices LOSA Report LOSA Raw Data • Phase of flight and overall text narratives • Listing and coding of every threat and error observed • Responses from an in-flight pilot interview LOSA Deliverables: Diagnostic Findings

  12. Sample LOSA Analysis

  13. Raw Data: Narratives LOSA Flight #23 Error #1 A340 ARPT 401 to ARPT 204 (Crew #8) Phase of Flight: Takeoff/Climb Pilot Flying: CA Takeoff / Climb Narrative – Almost immediately after takeoff, the Captain started a turn up the river, and did not wait until 400 feet, per SOP. About 300 feet, captain reached over to MCP (called FCU on Airbus) and selected Flight Path Angle. He did not tell FO of this selection, which is very unusual of a setting. In fact, the FO later asked me if I had ever seen anyone use this selection and stated that he has never seen it in his 2 years of being on the Airbus. Because the Captain did not tell the F/O what he was doing, when the FO noticed this setting, he reselected the more common mode (Heading). Captain then reached over and then reselected Flight Path Angle. This led to a brief discussion of whether of not the aircraft would capture NAV in this setting. As a result of all of this, the FO missed ATC handoff to departure control, a call that was not made until the aircraft leveled off at 5000 feet. Both Captain and FO were involved in heads-down stuff as the aircraft leveled off. If the autopilot would not have leveled off, then the crew would have not caught it, because they were doing extraneous things. (FO, programming MCDU; Captain, looking at weather).

  14. Raw Data: Threats LOSA Flight #:50Threat #2B737-700AUS / DFW Threat Description Threat: runway switch. We had expected and had briefed 17Left from the ATIS broadcast. No reason for the switch given by ATC. Threat Management Description Scramble to brief new runway plus linked to error of not finding the proper approach chart and automation error of not setting up computer for the landing runway. FO left the approach and arrival route for 17Left in the FMC instead of the new runway. Based on this the aircraft reached DIETZ and started a turn to heading 350 which was still the active route instead of the 300 heading assigned. Phase of Flight Des/App/Land Threat Type ATC Threat Code ATC runway change Threat Outcome: Linked to Flight Crew Error

  15. Summary - LOSA • Primary features • Purely diagnostic – no solutions – “flight operations health check” • Captures system safety and pilot performance strengths and weaknesses in normal operations using observations • Analysis • Aggregated TEM data provides a safety snapshot of flight operations on a typical day • Organizational change • Measure the effectiveness safety solutions with a follow-up LOSA

  16. LOSA Case Study One • Two LOSA’s (3 years apart) • 1st LOSA results – An example of targets identified: • Undesired number of “below standard” leadership ratings • Problems associated with checklist protocol / usage • High number of unstable approaches • Low rate of error trapping (errors detected and managed)

  17. LOSA Case Study One • Airline response to data: • Formed committees to address targets for enhancement • Implemented a leadership training module in recurrent • Revised checklist philosophy • Established bottom lines for unstable approaches • Implemented error management training course for all pilots • Check airmen were trained in evaluating error management

  18. LOSA Case Study One • 2nd LOSA results • Captain leadership scores significantly improved • 40% reduction in checklist errors • 77% reduction in unstable approaches below 1000 ft. • FOQA (QAR) showed similar trends • Two fold increase in error trapping rates • Change in the check culture to evaluate error management • Increase management “buy-in” on the safety change process

  19. LOSA Case Study Two • 1st LOSA results - Targets identified: • Improve energy management issues during descent/approach/land • 18% of flights with a speed deviation undesired aircraft state • Lower the prevalence of unstable approaches • 11% of flights with an unstable approach • Lower prevalence of checklist errors • 50% of flights with a checklist error • Improve monitor/cross-checking performance • 38% of errors went undetected (not responded to by the flight crew) • Improve ATC threat management • 22% of ATC threats were mismanaged - linked to flight crew error

  20. LOSA Case Study Two Proactive safety change ……. Over one year, approx 182,500 flights, 4% rate = 7,300 unstable – much better than 11% rate = 20,075 unstable

  21. Convergent Validity: LOSA and ASAP • Why would an airline do a LOSA when ASAP (incident reporting) captures the same things? • ASAP – collected when an event occurs • LOSA – collected independent of event • By focusing on LOSA data & implementing effective solutions, airlines can should be able to lower their potential for incidents / accidents “Well, James, very nice information. We see a lot of the same things in our ASAP database (voluntary incident reporting)” - Quote from an ASAP Project manager

  22. LOSA Operating Characteristics

  23. Angel Performance Natural Performance - + Pilot trust - + LOSA data quality LOSA Observer Secret to LOSA Success LOSA is not for all airlines - Its success depends on pilot trust Regulator Check Airman Nobody Low pilot trust = Low quality data because there will be no differentiation between LOSA and proficiency checks by regulators and check airmen

  24. Jumpseat observations during normal operations Anonymous, confidential, and non-punitive data collection Voluntary crew participation Trusted and trained observers Joint management / union sponsorship Systematic observation instrument Secure data collection repository Data verification roundtables Data-derived targets for enhancement Feedback of results to line pilots LOSA: How to Gain Pilot Trust LOSA is defined by 10 operating characteristics

  25. LOSA Operating Characteristics • Jumpseat observations during normal operations • Routine flights only • No line checks or training flights • Anonymous, confidential, and non-punitive data collection • No names, flight numbers, or other identifying information • Observer identity kept anonymous • Data only used for safety purposes – no disciplinary action • Voluntary crew participation • Flight crews have the right to decline a LOSA observation • Typical denial rate is very low – 1 per 100 flights

  26. LOSA Operating Characteristics • Trusted and trained observers • Trust • Selection process – management / union list of candidates • Mostly regular pilots - Captains and First Officers • Training • Five days – ground school (2), test observations (2) & recalibration (1) • Curriculum • LOSA protocol • How to write quality narratives • TEM recognition, recording and coding

  27. LOSA Operating Characteristics • Joint management / union sponsorship • Steering committee – Flight Ops, Training, Safety and Union • Symbolized with a signed agreement and sent to all pilots • Systematic observation instrument – TEM based • Observers only record the TEM events they see – no judgments • Secure data collection repository • Third party or pilot association gate keeper • Pilots must believe that observations will not be “misplaced”

  28. LOSA Operating Characteristics • Data verification roundtables • TEM data checked for coding accuracy and consistency with SOP • On completion, data analysis begins • Data-derived targets for enhancement • Serve as benchmarks for organizational change • “Measure, change, measure again” approach • Feedback of results to line pilots • Results andinformation on how airline management intends to respond with organizational change

  29. To ensure standardization, LOSA must have all ten operating characteristics If less than ten, the project needs another acronym Endorsed by: ICAO FAA (pending advisory circular) IATA UT TLC IFALPA US ALPA LOSA Defined

  30. Frequently Asked Questions

  31. LOSA Cost? • Depends on a number of factors – indirect and direct costs • Number of observations to be collected • Number of external and internal observers • Depth of data analysis • Insurance discounts? • External provider fee – LOSA Collaborative and others • Some examples of “misleading” efforts to lower costs • Observations only – no roundtables or data analysis • Operating crew as their own observers / self report • Video in the cockpit / analyzed by one retired pilot in the airline • These projects MIGHT have value but they are not LOSA

  32. Length and Size of LOSA? • Project Length • Average 8 months from planning to a final report • Airline – 75% of time spent on planning • LOSA Collaborative • 30% of time spent on planning, observer training, data collection • 70% spent on data verification, analysis and final report • Project Size – Want to extrapolate to everyday operations • Weighted sample to match daily departure rates per fleet • Minimum number of observations per fleet – 40 observations • Cap observers to no more than 15 observations to prevent observer fatigue

  33. Continuous or Snapshot LOSA? • Recommended: Snapshot approach with a sample of observations • Continuous LOSA – more cost, questionable value over snapshot LOSA • A snapshot approach is more manageable • There is a start and end for data collection that marks a point in time to derive targets and get to the real work of making changes • Snapshots can complement other continuous safety efforts such as QAR or incident reporting

  34. How Realistic is LOSA Data? • Doesn’t observers change the way flight crews behave? • If they do, then LOSA results are “underestimates” of what actually occurs in normal operations • If pilots “fake good” in front an observer, explain……. • Average three errors per flight – 25% are mismanaged • Average 32% of flights have an undesired aircraft state • Common to observe substandard CRM or procedural violations • Pilots calling girlfriends during taxi-out • Offering cigarettes to LOSA observers in the cockpit

  35. What about observer bias? • Major sources of observer bias • Sensitivity – Observers are more sensitive to certain areas of flight operations (ex, human factor experts that only see human factor problems) • Proficiency - Lack of TEM knowledge to correctly collect data • Fatigue – Observer burnout = less data collected = less precise • Some data quality controls • Use many different observers (Captains, First Officers, instructors, external) • Cap the number of observations per observer • Focus observers on gathering “just the facts” – no judgment of causes • Test for TEM knowledge during training backed up with data verification

  36. Biggest Threats to LOSA Success? • LOSA mutations and people calling or thinking its LOSA • Senior management not briefed and/or prepared for the possibility of undesirable results • Results not communicated to pilots – affects pilot acceptance for the second LOSA • Unmotivated observers • No safety change process in place to act on LOSA findings

  37. More Information about LOSA? • ICAO LOSA Handbook – Document 9803 • ICAO Journal – LOSA Special Edition (vol. 57, no.4, 2002) • UT website (www.psy.utexas.edu/HumanFactors) • FAA LOSA Advisory Circular (Draft) – Pending 2006 • Klinect Ph.D. dissertation - Academic demonstration of LOSA (available on request by e-mail) • Research papers on TEM and LOSA

  38. Audience Question and Answer