Model Governance Industry Evolution Beyond Model Accuracy

1. Model Governance Industry EvolutionBeyond Model Accuracy Bill Cember, FSA, MAAA

2. Disclaimer • Content in this presentation represent the views of the author. They do not necessarily reflect the views of ASNY, Prudential, the SOA, or other organizations

3. Agenda • Models and Model Governance • Guiding Principles – FAST • Cast Studies • Takeaways

4. Models and Model Governance • “A model is a quantitative method, system, or approach used to calculate or estimate value or risk that impact Prudential’s financial statements and/or assist in decision-making. A model consists of three fundamental components: (a) inputs and/or assumptions, (b) calculation routines, and (c) outputs and their adjustments. Models transform given inputs and/or assumptions into outputs with some degree of complexity and uncertainty.” - Prudential’s Model Risk Management Group

5. Models and Model Governance • “[T]he term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.” - SR-11

6. Guiding Principles - FAST • FAST • Flexible • Accurate • Standardized • Testable

7. Guiding Principles - Flexible • Flexibility = How easy it is to change your model when requirements change • Why it’s important: • If it’s hard to change a model, then the model won’t get improved. • More complicated changes to models not only take up more resources but are also more error prone • When models are hard to change, they will more easily brea inadvertantly

8. Guiding Principles - Flexible

9. Guiding Principles - Standardized • Standards = Common language that users and developers of the model can speak • Less Important what the standard is than a standard exists

10. Guiding Principles - Standardized

11. Guiding Principles - Testable

12. Guiding Principles - Testable

13. Guiding Principles - Accuracy • Accuracy is a line not a dot • New functionality can work when it is initially implemented but break at later dates when new “cases” arise • Simple ≠ Immune from errors • Developers should expect and be expected to make errors. All changes no matter how simple should be independently tested before being brought into production

14. Case Study • Your inforce has repeat policy numbers (that are legitimately different policies). How do you handle this?

15. Case Study • You want to run sensitivities on your model assumptions. Which of the approaches would you take to build this out? • Don’t build out anything. Run the model with new assumption tables ad hoc • Build out new assumption tables for the sensitivities. Develop model functionality to toggle which assumptions you want to use • Rebuild (if necessary) your original assumption tables such that sensitivities can be inputted as multipliers to the original tables. Develop model functionality to toggle whether to apply multipliers to the assumptions

16. Takeaways • Accuracy is a line not a dot. A model being right today does not mean it will be right tomorrow. • Model governance is more than just testing. • Develop your models in a FAST way to make your models more robust to change and easily testable to ensure changes are correct