making sense out of metrics l.
Skip this Video
Loading SlideShow in 5 Seconds..
Making Sense out of Metrics PowerPoint Presentation
Download Presentation
Making Sense out of Metrics

Loading in 2 Seconds...

play fullscreen
1 / 26

Making Sense out of Metrics - PowerPoint PPT Presentation

  • Uploaded on

Making Sense out of Metrics. ISDA / PRMIA 17 th August 2004. In 2002………. A global survey of 76 banks on the existence of formal KRI programs:. Acknowledgement Raft International Limited. What is a KRI?. The number of fails has increased by 2% Is this a KRI?. Some definitions…….

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Making Sense out of Metrics' - ostinmannual

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
making sense out of metrics

Making Sense out of Metrics


17th August 2004

in 2002
In 2002………
  • A global survey of 76 banks on the existence of formal KRI programs:

Acknowledgement Raft International Limited

what is a kri
What is a KRI?

The number of fails has increased by 2%

Is this a KRI?

some definitions
Some definitions……
  • Metric – something observed or calculated that is used to show the presence or state of a condition or trend; an instrument or gauge that measures something and registers the measurement; something such as a light, sign, or pointer that gives information, for example about which direction to follow
  • KRI – Key Risk Indicator
  • KPI – Key Performance Indicator
  • KCI – Key Control Indicator
  • KMI – Key Management Indicator

Overall, prefer the general term “indicator”

what is a risk indicator
What is a Risk Indicator?
  • Risk indicators are usually monitored over time
  • Late trade processing in Bank X in May 2004:

London 9%, New York 10%, Singapore 9%

  • In which branch

do we have a


what is a risk indicator6
What is a Risk Indicator?

absolute numbers

  • So, Singapore is the problem!
  • Is it?

Data without context may not expose the entire problem!

the choice from the multitude
The Choice from the Multitude
  • A typical operation can identify hundreds of indicators
  • Some are risk, others performance indicators
  • Indicators’ relevance and weight change over time
  • Some indicators are meaningless on their own

Graphics adapted from Reason, J.: “Managing the Risks of Organizational Accidents", Aldershot: Ashgate, 1997

other factors
Other factors
  • If a ratings agency is to rate operational exposure, how will they compare different organisations?
  • How will regulators evaluate the effectiveness of different organisation’s operational risk capabilities?
  • How does a business unit provide senior management quality information?
  • Can the organisation use operational metrics to provide stakeholder information?
but kris have been disappointing
But, KRIs have been disappointing…
  • No means of consistently relate the occurrence of loss events and the location of problems/situations
  • No means of classifying types of KRIs
  • Plenty of data but no idea of its relevance
  • No way to determine relevance
  • No observable best practice
  • No means of comparison, either internally or externally
the solution
The solution?
  • An organisation needs a common language or framework which identifies areas of exposure and then allows…..
    • Metrics to be identified to measure, monitor and manage those exposures
    • Data on losses, near misses and control failures to be recorded
    • Ongoing assessment of the exposure
    • Performance measurement around the exposure, including use of capital
kri study background
KRI Study - Background
  • Recognising the need, RiskBusiness developed a strawman framework for identifying risk points within an institution
  • RMA and RiskBusiness co-sponsored Part I of the Study to ascertain the feasibility of using the framework as a KRI framework
  • Seven institutions tested the framework in a risk mapping exercise
initial participants
Initial Participants
  • 7 banks participated in Part I:
    • Citigroup
    • Deutsche Bank
    • Dresdner Kleinwort Wasserstein
    • JP Morgan Chase
    • KeyCorp
    • Royal Bank of Canada
    • State Street
  • ANZ and Abbey then joined to form the Study Steering Group for Part II
the study
The Study
  • Part I – Proof of Concept
  • Part II – 3 primary activities :
    • Broaden participation, transfer experience, build industry risk profile
    • Define KRI Library
    • Develop detailed specifications
  • Future :
    • Industry benchmarking
    • Extend participation further
  • Initial risk maps were evaluated to establish “most risky” risk categories and business functions
  • Compared these to a similar evaluation of the QIS 3 data and to the complete Fitch Risk First database
  • Broad correlation, taking into account the nature of the two data sources
participant risk map deviations
Participant risk map deviations
  • Based on risk category:
participant risk map deviations17
Participant risk map deviations
  • Based on business function:
participants in part ii

ABN Amro


Acleda Bank

Alliance and Leicester


Bank Austria Creditanstaldt

Bank Julius Bäer

Bank of America

Bank Rakvat Indonesia

Bank Vontobel

BNP Paribas

Byblos Bank

Capital One



De Lage Landen (sub of Rabobank)

Deutsche Bank†

Dresdner Kleinwort Wasserstein†

Erste Bank


Federation de Caisses Desjardins du Quebec


Halifax Bank of Scotland


Huntington National Bank


JP Morgan Chase†


Kookmin Bank

Macquarie Bank

Mizuho International

National Australia Group

National Bank of Canada

Nomura International

Northern Trust

People’s Bank

Royal Bank of Canada†

Royal Bank of Scotland


San Paolo IMI

SE Banken

Southwest Bank of Texas

Standard Bank of South Africa

State Street†

Sumitomo Mitsui

TD Financial Group

Woori Bank

Washington Mutual

Participants in Part II

† = Lead Participant

defining indicators
Defining Indicators
  • Identify candidates for each risk point
  • Evaluate candidates against qualifying criteria (effectiveness, comparability, ease of use/collection)
  • Agree descriptions for each qualifier
  • Prioritise nominated indicators
  • Participant comment and review
  • Generate detailed specifications, stored in KRI Library at
considering each risk point
Considering each risk point…..

Before the problem has been generated

Look for ‘Early

Warning Flags’

After the problem has been generated

Identify ‘Problems

in Progress’

Potential loss events, include

events that could lead this or another

institution to loss

Near misses, loss = 0


‘Historical Events’

0 < Losses < threshold

Losses > threshold

issues for consideration
Issues for consideration
  • What is a KRI...KPI…KCI…or MIS?
    • An indicator can perform multiple roles depending on who is using it
  • What about scaling and aggregation?
    • Do we scale then aggregate, or vice versa?
  • How many indicators should a firm be monitoring?
    • The quest for the “Magic 10” – the KRI Library has 1,600 indicators
issues for consideration22
Issues for consideration
  • Top-down versus Bottom-Up
    • Operations develop metrics for ongoing use, Management want information
  • Combinations and clusters of indicators
    • Experience has demonstrated that in many cases, it is groups of indicators which will provide the best management information
    • Staff Turnover and Transaction Volumes and Error Rates and ……
next steps benchmarking
Next Steps - Benchmarking
  • Selected indicators will be driven totally by broad participant agreement
  • Consists of participants delivering KRI data to centralised function :
    • Data will be anonymous
    • Data will be collated, analysed and benchmark values calculated
  • Participants have access to benchmarks for comparative purposes
  • First submission expected in Q2, 2005
next steps kri library
Next Steps – KRI Library
  • Next intake of participants into KRI Library starts in October 2004 – currently have 68 additional firms wishing to join
  • Insurance KRI Study starts during Q4 2004
  • Ongoing maintenance and extension to the Library
in summary
In Summary…..
  • A well-developed and structured indicator program can deliver quality management information and could possibly be used as an adjustor to capital…or at least as a measure of efficient and effective use
  • Common language and standardisation is imperative
  • The indicator program must deliver value at all levels
contact details
Contact details
  • RiskBusiness International Limited
    • URL :
    • Study URL :
  • Mike Finlay, Managing Director – Europe, Asia, Australia and Africa
    • Telephone : +44 7721 969 224
    • E-mail :