Line operations safety audit losa a practical overview
Advertisement
This presentation is the property of its rightful owner.
1 / 38

Practical overview of the LOSA process PowerPoint PPT Presentation

Malaysia Airlines, Kuala Lumpur - September 13-14, 2005 ... LOSA is not for all airlines - Its success depends on pilot trust ...

Download Presentation

Practical overview of the LOSA process

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Line operations safety audit losa a practical overview

The University of Texas

Human Factors Research Project

Third ICAO-IATA LOSA & TEM Conference

Malaysia Airlines, Kuala Lumpur - September 13-14, 2005

Line Operations Safety Audit (LOSA): A Practical Overview

James Klinect and Patrick Murray

The University of Texas / The LOSA Collaborative


Introduction james klinect

Introduction: James Klinect

  • University of Texas (UT) – LOSA research for the industry

    • Project Coordinator – TEM & LOSA

    • Director: Bob Helmreich

  • LOSA Collaborative – LOSA implementation for airlines

    • Role: Founder/President

    • De-identified data to University of Texas

  • Disclaimer: Not an commercial for the LOSA Collaborative

    • Sharing of mistakes made and lessons learned

    • Provide a enough information to evaluate LOSA and possibly implement it themselves (ex, Varig, Lan Chile,….)


Losa airlines 1996 2005

AeroMexico

Asiana Airlines

Alaska Airlines

Air New Zealand

Braathens

Cathay Pacific

China Airlines

COPA

LOSA Airlines: 1996-2005

Since 1996, 4500+ observations, 25 airlines & 11 countries

Continental Airlines

Continental Express

Continental Micronesia

Delta Airlines

EVA Air

Frontier Airlines

LACSA

QANTAS

Malaysia Airlines

Mt. Cook

Regional Express

Singapore Airlines

Silk Air

TACA / TACA Peru

UNI Air

US Airways


Presentation outline

Presentation Outline

  • LOSA Rationale / Deliverables

    • Why LOSA and what do you get?

  • LOSA Operating Characteristics

    • What is the definition of LOSA and how does it work?

  • LOSA E-mail Inbox

    • What are the most frequently asked LOSA questions?

  • Audience Question and Answer


Losa rationale deliverables

LOSA Rationale / Deliverables


Aviation safety envelope

Safety

Aviation Safety Envelope

Accidents

Incidents

Normal Operations


Safety data coverage

Voluntary Incident Reports

LOSA

QAR

Mandatory Incident Reports

Accident Investigation

Safety Data Coverage

Accidents

Incidents

Normal Operations


Aviation safety envelope1

Aviation Safety Envelope

Accidents

Incidents

Normal Operations

LOSA Rationale

Proactive snapshot of system / crew performance strengthsand weaknesses

in normal operations (just like a health checkup)


Losa measurement challenge

LOSA Measurement Challenge

  • Challenge - What type of observational measure best captures safety performance data in normal operations?

  • Concerns

    • Observer overload – Can’t capture everything

    • Data analyst overload – Must have a roadmap for interpretation

    • Manager overload – Data need face value and point to safety issues

  • Solution – Threat and Error Management (TEM)


Primary losa measures tem

Error Management

1

Manage operational complexity

Threat Management

LOSA Measures

2

Manage their own errors

3

Manage aircraft deviations

Undesired Aircraft State Management

Primary LOSA Measures: TEM

What do flight crews do to safely fly from A to B?

Where’s CRM and human factors? Non-technical skills are one of many contributors to effective/ineffective TEM performance


Losa deliverables diagnostic findings

Most frequent and mismanaged threats

Most frequent and mismanaged errors

Fleet and operational differences

Comparison to other airlines on LOSA safety indices

LOSA Report

LOSA Raw Data

  • Phase of flight and overall text narratives

  • Listing and coding of every threat and error observed

  • Responses from an in-flight pilot interview

LOSA Deliverables: Diagnostic Findings


Sample losa analysis

Sample LOSA Analysis


Raw data narratives

Raw Data: Narratives

LOSA Flight #23Error #1A340 ARPT 401 to ARPT 204(Crew #8)

Phase of Flight: Takeoff/ClimbPilot Flying: CA

Takeoff / Climb Narrative – Almost immediately after takeoff, the Captain started a turn up the river, and did not wait until 400 feet, per SOP. About 300 feet, captain reached over to MCP (called FCU on Airbus) and selected Flight Path Angle. He did not tell FO of this selection, which is very unusual of a setting. In fact, the FO later asked me if I had ever seen anyone use this selection and stated that he has never seen it in his 2 years of being on the Airbus. Because the Captain did not tell the F/O what he was doing, when the FO noticed this setting, he reselected the more common mode (Heading). Captain then reached over and then reselected Flight Path Angle. This led to a brief discussion of whether of not the aircraft would capture NAV in this setting. As a result of all of this, the FO missed ATC handoff to departure control, a call that was not made until the aircraft leveled off at 5000 feet. Both Captain and FO were involved in heads-down stuff as the aircraft leveled off. If the autopilot would not have leveled off, then the crew would have not caught it, because they were doing extraneous things. (FO, programming MCDU; Captain, looking at weather).


Raw data threats

Raw Data: Threats

LOSA Flight #:50Threat #2B737-700AUS / DFW

Threat Description

Threat: runway switch. We had expected and had briefed 17Left from the ATIS broadcast. No reason for the switch given by ATC.

Threat Management Description

Scramble to brief new runway plus linked to error of not finding the proper approach chart and automation error of not setting up computer for the landing runway. FO left the approach and arrival route for 17Left in the FMC instead of the new runway. Based on this the aircraft reached DIETZ and started a turn to heading 350 which was still the active route instead of the 300 heading assigned.

Phase of FlightDes/App/Land

Threat TypeATC

Threat CodeATC runway change

Threat Outcome:Linked to Flight Crew Error


Summary losa

Summary - LOSA

  • Primary features

    • Purely diagnostic – no solutions – “flight operations health check”

    • Captures system safety and pilot performance strengths and weaknesses in normal operations using observations

  • Analysis

    • Aggregated TEM data provides a safety snapshot of flight operations on a typical day

  • Organizational change

    • Measure the effectiveness safety solutions with a follow-up LOSA


Losa case study one

LOSA Case Study One

  • Two LOSA’s (3 years apart)

  • 1st LOSA results – An example of targets identified:

    • Undesired number of “below standard” leadership ratings

    • Problems associated with checklist protocol / usage

    • High number of unstable approaches

    • Low rate of error trapping (errors detected and managed)


Losa case study one1

LOSA Case Study One

  • Airline response to data:

    • Formed committees to address targets for enhancement

    • Implemented a leadership training module in recurrent

    • Revised checklist philosophy

    • Established bottom lines for unstable approaches

    • Implemented error management training course for all pilots

    • Check airmen were trained in evaluating error management


Losa case study one2

LOSA Case Study One

  • 2nd LOSA results

    • Captain leadership scores significantly improved

    • 40% reduction in checklist errors

    • 77% reduction in unstable approaches below 1000 ft.

      • FOQA (QAR) showed similar trends

    • Two fold increase in error trapping rates

    • Change in the check culture to evaluate error management

    • Increase management “buy-in” on the safety change process


Losa case study two

LOSA Case Study Two

  • 1st LOSA results - Targets identified:

    • Improve energy management issues during descent/approach/land

      • 18% of flights with a speed deviation undesired aircraft state

    • Lower the prevalence of unstable approaches

      • 11% of flights with an unstable approach

    • Lower prevalence of checklist errors

      • 50% of flights with a checklist error

    • Improve monitor/cross-checking performance

      • 38% of errors went undetected (not responded to by the flight crew)

    • Improve ATC threat management

      • 22% of ATC threats were mismanaged - linked to flight crew error


Losa case study two1

LOSA Case Study Two

Proactive safety change …….

Over one year, approx 182,500 flights, 4% rate = 7,300 unstable – much better than 11% rate = 20,075 unstable


Convergent validity losa and asap

Convergent Validity: LOSA and ASAP

  • Why would an airline do a LOSA when ASAP (incident reporting) captures the same things?

    • ASAP – collected when an event occurs

    • LOSA – collected independent of event

  • By focusing on LOSA data & implementing effective solutions, airlines can should be able to lower their potential for incidents / accidents

“Well, James, very nice information. We see a lot of the same things in our ASAP database (voluntary incident reporting)”

- Quote from an ASAP Project manager


Losa operating characteristics

LOSA Operating Characteristics


Secret to losa success

Angel

Performance

Natural

Performance

-

+

Pilot trust

-

+

LOSA data quality

LOSA Observer

Secret to LOSA Success

LOSA is not for all airlines - Its success depends on pilot trust

Regulator

Check Airman

Nobody

Low pilot trust = Low quality data because there will be no differentiation between LOSA and proficiency checks by regulators and check airmen


Losa how to gain pilot trust

Jumpseat observations during normal operations

Anonymous, confidential, and non-punitive data collection

Voluntary crew participation

Trusted and trained observers

Joint management / union sponsorship

Systematic observation instrument

Secure data collection repository

Data verification roundtables

Data-derived targets for enhancement

Feedback of results to line pilots

LOSA: How to Gain Pilot Trust

LOSA is defined by 10 operating characteristics


Losa operating characteristics1

LOSA Operating Characteristics

  • Jumpseat observations during normal operations

    • Routine flights only

    • No line checks or training flights

  • Anonymous, confidential, and non-punitive data collection

    • No names, flight numbers, or other identifying information

    • Observer identity kept anonymous

    • Data only used for safety purposes – no disciplinary action

  • Voluntary crew participation

    • Flight crews have the right to decline a LOSA observation

    • Typical denial rate is very low – 1 per 100 flights


Losa operating characteristics2

LOSA Operating Characteristics

  • Trusted and trained observers

    • Trust

      • Selection process – management / union list of candidates

      • Mostly regular pilots - Captains and First Officers

    • Training

      • Five days – ground school (2), test observations (2) & recalibration (1)

      • Curriculum

        • LOSA protocol

        • How to write quality narratives

        • TEM recognition, recording and coding


Losa operating characteristics3

LOSA Operating Characteristics

  • Joint management / union sponsorship

    • Steering committee – Flight Ops, Training, Safety and Union

    • Symbolized with a signed agreement and sent to all pilots

  • Systematic observation instrument – TEM based

    • Observers only record the TEM events they see – no judgments

  • Secure data collection repository

    • Third party or pilot association gate keeper

    • Pilots must believe that observations will not be “misplaced”


Losa operating characteristics4

LOSA Operating Characteristics

  • Data verification roundtables

    • TEM data checked for coding accuracy and consistency with SOP

    • On completion, data analysis begins

  • Data-derived targets for enhancement

    • Serve as benchmarks for organizational change

    • “Measure, change, measure again” approach

  • Feedback of results to line pilots

    • Results andinformation on how airline management intends to respond with organizational change


Losa defined

To ensure standardization, LOSA must have all ten operating characteristics

If less than ten, the project needs another acronym

Endorsed by:

ICAO

FAA (pending advisory circular)

IATA

UT

TLC

IFALPA

US ALPA

LOSA Defined


Frequently asked questions

Frequently Asked Questions


Losa cost

LOSA Cost?

  • Depends on a number of factors – indirect and direct costs

    • Number of observations to be collected

    • Number of external and internal observers

    • Depth of data analysis

    • Insurance discounts?

    • External provider fee – LOSA Collaborative and others

  • Some examples of “misleading” efforts to lower costs

    • Observations only – no roundtables or data analysis

    • Operating crew as their own observers / self report

    • Video in the cockpit / analyzed by one retired pilot in the airline

    • These projects MIGHT have value but they are not LOSA


Length and size of losa

Length and Size of LOSA?

  • Project Length

    • Average 8 months from planning to a final report

    • Airline – 75% of time spent on planning

    • LOSA Collaborative

      • 30% of time spent on planning, observer training, data collection

      • 70% spent on data verification, analysis and final report

  • Project Size – Want to extrapolate to everyday operations

    • Weighted sample to match daily departure rates per fleet

    • Minimum number of observations per fleet – 40 observations

    • Cap observers to no more than 15 observations to prevent observer fatigue


Continuous or snapshot losa

Continuous or Snapshot LOSA?

  • Recommended: Snapshot approach with a sample of observations

  • Continuous LOSA – more cost, questionable value over snapshot LOSA

  • A snapshot approach is more manageable

    • There is a start and end for data collection that marks a point in time to derive targets and get to the real work of making changes

  • Snapshots can complement other continuous safety efforts such as QAR or incident reporting


How realistic is losa data

How Realistic is LOSA Data?

  • Doesn’t observers change the way flight crews behave?

    • If they do, then LOSA results are “underestimates” of what actually occurs in normal operations

  • If pilots “fake good” in front an observer, explain…….

    • Average three errors per flight – 25% are mismanaged

    • Average 32% of flights have an undesired aircraft state

    • Common to observe substandard CRM or procedural violations

    • Pilots calling girlfriends during taxi-out

    • Offering cigarettes to LOSA observers in the cockpit


What about observer bias

What about observer bias?

  • Major sources of observer bias

    • Sensitivity – Observers are more sensitive to certain areas of flight operations (ex, human factor experts that only see human factor problems)

    • Proficiency - Lack of TEM knowledge to correctly collect data

    • Fatigue – Observer burnout = less data collected = less precise

  • Some data quality controls

    • Use many different observers (Captains, First Officers, instructors, external)

    • Cap the number of observations per observer

    • Focus observers on gathering “just the facts” – no judgment of causes

    • Test for TEM knowledge during training backed up with data verification


Biggest threats to losa success

Biggest Threats to LOSA Success?

  • LOSA mutations and people calling or thinking its LOSA

  • Senior management not briefed and/or prepared for the possibility of undesirable results

  • Results not communicated to pilots – affects pilot acceptance for the second LOSA

  • Unmotivated observers

  • No safety change process in place to act on LOSA findings


More information about losa

More Information about LOSA?

  • ICAO LOSA Handbook – Document 9803

  • ICAO Journal – LOSA Special Edition (vol. 57, no.4, 2002)

  • UT website (www.psy.utexas.edu/HumanFactors)

    • FAA LOSA Advisory Circular (Draft) – Pending 2006

    • Klinect Ph.D. dissertation - Academic demonstration of LOSA (available on request by e-mail)

    • Research papers on TEM and LOSA


Audience question and answer

Audience Question and Answer


  • Login