The advanced irb forum monday 23 june l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 27

THE ADVANCED IRB FORUM Monday 23 June PowerPoint PPT Presentation


  • 130 Views
  • Uploaded on
  • Presentation posted in: General

THE ADVANCED IRB FORUM Monday 23 June. Kevin Ryan . AREAS TO BE COVERED. Issues in validating Internal Ratings systems How are supervisors reacting – emerging thoughts The FSA Consultative Paper Sections on Validation, External Models & Data, Data Quality, Assessment Horizon & PD estimation.

Download Presentation

THE ADVANCED IRB FORUM Monday 23 June

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The advanced irb forum monday 23 june l.jpg

THE ADVANCED IRB FORUMMonday 23 June

Kevin Ryan


Areas to be covered l.jpg

AREAS TO BE COVERED

  • Issues in validating Internal Ratings systems

  • How are supervisors reacting – emerging thoughts

  • The FSA Consultative Paper

    • Sections on Validation, External Models & Data, Data Quality, Assessment Horizon & PD estimation


Not to be covered l.jpg

NOT TO BE COVERED

Detailed ‘how to do it’ to get your IRB systems

approved by the FSA

  • We do not know enough at this stage

    • “We recognise a lot of work remains in this complex area with many challenges for firms and us”

  • Responsibility of firms to validate their rating systems

    • Firms ‘validate’; supervisors ‘approve’ (or certify)


Stylised supervisors view of rating systems l.jpg

STYLISED SUPERVISORS’ VIEW OF RATING SYSTEMS

  • Data and validation a challenge!

  • Good work has been done, but not been part of the mainstream activity of the firms, so

    • Even in the best firms, few people understand the issues

    • Knowledge and oversight by senior management and control functions quite low

    • Resources for improvements can be difficult to get, so trade-offs made

    • Limited appetite for fresh thinking or challenging assumptions

  • Appetite for short cut solutions, eg agency ratings


A model is proved by its performance l.jpg

A MODEL IS PROVED BY ITS PERFORMANCE ?

SMALL SAMPLE SIZE

In the common binomial test, construct a

confidence interval around estimated PD

  • Suppose a PD estimate is 100bp, and you want to be 95% confident that actual PD is between 80 and 120bp


A model is proved by its performance6 l.jpg

A MODEL IS PROVED BY ITS PERFORMANCE ?

SMALL SAMPLE SIZE

In the common binomial test, construct a

confidence interval around estimated PD

  • Suppose a PD estimate is 100bp, and you want to be 95% confident that actual PD is between 80 and 120bp

    You need 9500 borrowers in that grade


A model is proved by its performance7 l.jpg

A MODEL IS PROVED BY ITS PERFORMANCE ?

SMALL SAMPLE SIZE

Using the common binomial test, what levels of

true PD are consistent with zero observed

defaults, at a 95% confidence level?

  • If you have 500 borrowers in a grade, any PD up to 1.25%

  • If you have 80 in a grade, a PD as high as 6% or 7%

    (Binomial test doesn’t work if PD x Borrowers below 5)


A model is proved by its performance8 l.jpg

A MODEL IS PROVED BY ITS PERFORMANCE ?

DEFAULTS ARE CORRELATED

  • The distribution you can expect from defaults is many periods with actual defaults below the mean, and fewer periods (typically clustered together) with actual defaults well above the mean

    IMPLICATIONS

  • Confidence bounds wider than binomial

  • More difficult to interpret


A model is proved by its performance9 l.jpg

A MODEL IS PROVED BY ITS PERFORMANCE ?

IMPLICATIONS

  • “Regardless of the efforts used by banks, …there will still remain some portfolios where there will likely never be sufficient default data to calibrate PDs with any degree of statistical significance”

    Internal Ratings Validation Study

  • “We’re going to need to be pragmatic for some years to come”


What can be done l.jpg

WHAT CAN BE DONE?

BROADER APPROACH TO VALIDATION, TO

SUPPLEMENT OUTCOMES ANALYSIS

  • Logic and conceptual soundness of the approach

  • Statistical testing prior to use

  • Monitoring of process – are the methods being applied as intended

  • Benchmarking – compare to relevant alternatives

    David Wright, FRB, 19 June


What can be done11 l.jpg

WHAT CAN BE DONE?

MORE ON OUTCOMES ANALYSIS, INCLUDING

  • More exploration and use of tolerance levels by firms, which requires

  • Better understanding of distribution of expected defaults

  • Supervisory interest in information content of transition matrices

  • What can be learnt from other industries?

    • Insurance experience of modeling rare events?


Fsa approach fsa cp l.jpg

FSA APPROACH – FSA CP

KEY POINTS

  • Accountability

    • Firms responsibility to validate & submit documentation with IRB application

    • Application to be signed by chief executive

      • We expect he will have similar questions to supervisors

  • Independence

    • Approved by senior committee

    • Independent staff to participate


Fsa approach fsa cp13 l.jpg

FSA APPROACH – FSA CP

KEY POINTS

  • Scope

    • To cover all portfolios, but depth will vary with significance

      • Materiality considerations

    • Methods may vary, especially by portfolio

      • If more reliance can be placed on back-testing we should need less additional evidence

      • If less reliance on back-testing, more additional evidence is needed


Fsa approach fsa cp14 l.jpg

FSA APPROACH – FSA CP

KEY POINTS

  • Scope

    • Must assess the accuracy of the overall output of the system

      • Not just the inputs

      • Will need to take account of overrides and judgmental adjustments to any underlying statistical models

    • Based on statistical analysis of rating system, related internal data, and third party and publicly available information


Fsa approach fsa cp15 l.jpg

FSA APPROACH – FSA CP

  • Coverage

    • Full documentation

    • Clarity on what rating system aiming to predict, and expected distributions

    • Take full account of adjustments between unbiased estimates and those used in regulatory capital calculation

      • Conservatism

      • Cyclical effects

      • Double default effects


Fsa approach fsa cp16 l.jpg

FSA APPROACH – FSA CP

  • Coverage

    • Clearly set out standards of control

    • Clearly set out limitations of approach

      • Assume that there will be candour

    • Include work to demonstrate both ability to rank into grades (discriminate) and estimate a PD etc (calibrate)

    • Include steps to be taken in event quantitative tests for power and accuracy are breached


Fsa approach fsa cp17 l.jpg

FSA APPROACH – FSA CP

  • Discrimination/ranking

    • Firm must justify that system shows a ‘high degree of power in line with industry norm for portfolios of that nature’

    • We do not specify what test should be used, eg Gini or other, or its level

      • But we consult on such a test

    • Expect firms to strive for best model they can, other things being equal


Fsa approach fsa cp18 l.jpg

FSA APPROACH – FSA CP

  • Discrimination/ranking - issues

    • Some see discrimination as key to model building

    • Some research that high discrimination needed to achieve accurate PD measures

    • But levels dependent on features of portfolio, number of defaults

    • High is good, but may be over-fitting

    • Firms must use targets, but reluctant to admit to them

    • Importance of expertise to interpret


Fsa approach fsa cp19 l.jpg

FSA APPROACH – FSA CP

  • Calibration

    • Standardised ‘scorecard’ proposed

    • Firm to justify its estimate against own historic experience and external sources

    • Take account of factors which may expect to lead to differences; eg conservatism, cycle effects

    • We do not specify tests for assessing accuracy

      • But we consult on such a test

    • If actuals not consistent with estimates, firms must justify differences and/or take steps to improve


Fsa approach fsa cp20 l.jpg

FSA APPROACH – FSA CP

  • Data quality

    • “Data quality was not seen as the major obstacle to validation in the long term”

      Internal Ratings Validation Study

    • Some evidence that cleaning the data produces more powerful models, while greater accuracy self evident

    • Missing defaults?

    • “We recognise that the data accuracy challenge for IRB is significant” FSA CP

    • “We propose quantifiable targets to cover completeness and accuracy that will rise over time”


Fsa approach fsa cp21 l.jpg

FSA APPROACH – FSA CP

  • External models and data

    • In principle supportive as can supplement internal data and models; also external vendors may operate to higher standards than firms

    • But external models carry explicit risk of black box, with limitations not known and application to inappropriate portfolios

    • Limit to how much vendors will reveal

    • We are trialing ‘vendor grid’ which would standardise information to be provided – aimed at reducing risks


Fsa approach fsa cp22 l.jpg

FSA APPROACH – FSA CP

  • Pooled data

    • In principle supportive as, like external data, can supplement internal experience, and without commercial confidentiality issues of external vendors

    • “Issues and challenges facing pooling initiatives

      • Consistent data definitions for all

      • Legal restrictions – data protection and confidentiality

      • Potential increase in systemic risk”

        Internal Ratings Validation Study


Fsa approach fsa cp23 l.jpg

FSA APPROACH – FSA CP

  • Expert judgment

    • “There is a feeling that not enough time has been spent on discussing acceptable validation techniques for these types of systems”

      Internal Ratings Validation Study

    • Will look to accommodate, but validation challenges increase

    • Are there some cases where it is not feasible to produce quantified estimates?

    • Or use ranking with conservative estimates of losses


Benchmarking l.jpg

BENCHMARKING

  • A complement to back-testing

  • Conceptually two types, although differences between them may be blurred in practice

    • Relative benchmarks – compare estimates between firms to identify outliers

    • Absolute benchmarks - compare with an external benchmark given some credibility

  • We propose some element of benchmarking in our proposed approaches to discrimination (industry standard) and calibration (comparison with external sources)


Benchmarking25 l.jpg

BENCHMARKING

POSSIBLE INITIATIVES

  • Private sector services to benchmark ratings and/or estimates

  • Some international interest in requiring firms to rate ‘test portfolios’

  • FSA considering targeted benchmarking at obligor level where back-testing difficult and amounts large

    • Very early stage of thinking but would welcome feedback

    • Could be run by industry, FSA or other body


Some validation challenges l.jpg

SOME VALIDATION CHALLENGES

  • Scale of task given possible number of models

    • Materiality to firm v materiality to the market

    • Importance of consistency

  • Need for firms and supervisors to increase expertise

    • Even statistical models require expert judgment

    • How much do supervisors need to know?

  • Can we give enough guidance to allow objective self-assessment?

  • What is the scope and appetite for improved standards?


Some validation challenges27 l.jpg

SOME VALIDATION CHALLENGES

  • What is the supervisory standard?

    • Can we identify a definite pass or a definite fail?

      • Benchmarks which must be beaten to qualify, or if beaten are sufficient

      • Implications for expert judgment approaches, or methods which ‘perform’ less well?

      • Other objectives may be avoidance of systemic risk, and allowing entry to stimulate competition

    • How do we incentivise firms to improve?

      • Rising standards or Pillar 2 requirements

    • Too much conservatism will take away incentives

  • LGD and EAD?


  • Login