The Business of Predictive Modeling

The Business ofPredictive Modeling December 17, 2013 Christine Hofbeck, FSA, MAAA Centroid Analytics, LLC

AGENDA PART I -- INTRODUCTION PART II – MODELING 101 (Basic Steps) PART III – “GOLDEN QUESTION” PART IV – OPERATIONAL CONSIDERATIONS

INTRODUCTION Predictive modelling [sic] is the process by which a model is created or chosen to try to best predict the probability of an outcome. -- Wikipedia In practice: VISUALIZE Customers Most Profitable Lines Identify patterns/ segment risks Develop business rules Improved decision making OTHER you are only limited by your creativity

Potential Applications OPTIMIZE Operational Efficiency Distribution Channels Claims Management Pricing / Reserves VISUALIZE Customers Most Profitable Lines or Products Target Marketing DATA MINIMIZE Risk Fraud OTHER you are only limited by your creativity

Potential Applications (Life) • Triage UW decisions; implement STP for (more) applicants • Decrease purchase of traditional UW requirements by determining when they may not be necessary • Identify & target customers more likely to buy • Identify customers more likely to lapse – intervene if profitable, allow unhealthies to lapse • Inforce book management • Identify most desirable agents • Smart customer handling

Predictors A predictive model is made up of a number of predictors (“independent variables”), which are data elements likely to influence future behavior or results (“dependent variable”). DON’T USE ONE VARIABLE DON’T USE ALL VARIABLES the mean predicts the future but doesn’t tell us why… (“underfit”) exactly replicates the past… cannot predict the future (“overfit”) SEEK PARSIMONY

BASIC STEPS (Modeling 101)

Define & Scope How will the results be used? For whom/what are we trying to predict this? (“unit of exposure”) Exactly what are we trying to predict? How long do we have to build? To implement? What is the budget? Consider IT, staff, data purchase, training, etc. Do we have the systems capacity to implement? Insource or outsource?

Data Prep * sometimes the most time intensive step of modeling • INTERNAL DATA • # years • accuracy • ability to access • primary key • EXTERNAL DATA • match rate • cost – to model • cost – to use • frequency of update MODELING DATASET Consider both expected & unexpected relationships – creativity in data exploration can be the key to your competitive edge!

Data Prep (cont’d) • COMBINE various data sources • CONVERT to desired exposure unit or format • CORRECT inaccurate data • INSPECT to remove variables: - Too many blank values that cannot be imputed - All/most values the same - Data cannot be relied upon - Data will not be captured going forward - Legal advice not to use • BUCKET(“bin”) values

Model Build (cont’d) UNIVARIATE ANALYSIS – test each variable one by one to see which ones may be predictive. MULTIVARIATE ANALYSIS – examine multiple variables in different groups to obtain the best, USABLE results – remember parsimony! INTERACTIONS – which variables can be combined into a “mega variable” to improve results (i.e., does 1+1 = 1.5? does 1+1 = 3?) Complicate the model (add variables, interactions) and simplify the model (remove variables, bin) to find the preferred combination.

Model Build (cont’d) Various tests can be used to determine variable inclusion: STATISTICAL JUDGMENT CONSISTENCY P-values Cramer’s V Confidence intervals Type III tests Of patterns - Over time Over random parts of a dataset Apply business knowledge to assess whether suggested relationships make sense

Model Validation ACTUAL vs. EXPECTED-- how close did we get? Generally, a subset of the data is withheld during the modeling process for validation: Model validation graphs are useful for communicating model performance to non-technical audiences. OUT OF TIME withhold most recent data OUT OF SAMPLE withhold randomly generated % of records

Model Validation – Sample Chart Outcome Decile

Implementation BUSINESS RULES What decisions will be made based on the prediction? May vary by location, business, rate group, etc. SYSTEM BUILD Scoring engine (collects data & calculates predictions) Decision tool (executes business rules) User interface TRAINING Anyone who will interact with the model must understand what it does and why

Review & Refine REPORTING How close did we get to the goal? How far did we exceed it? Multiple reporting packages required for varied audiences, for example: • Executives – highlights in aggregate by zone, business unit, product • Actuaries – detailed results by variable, state, rate group • Marketing – by broker/agent, location • Underwriting – by underwriter as a performance measure Frequency of update – weekly, monthly, quarterly, yearly? Method of calculation – automated? ad hoc?

Review & Refine (cont’d) MODEL UPDATES WHY? • As target customer is attained, characteristics of inforce book will change • Business goals/strategies may change • New data may become available • Tolerance for certain characteristics may change HOW? • Update current variable relativities (“recalibrate”) • Start over - search for more predictive variables (“recast”) HOW OFTEN?

Advantages of Modeling Over Traditional Approaches • Many additional and often unconventional variables may be examined • Modeling a particular variable controls for the effects of other included variables – we don’t risk double counting or attributing effects to the wrong variables • Traditional approaches segment data into smaller categories which impact credibility • Interactions are introduced The above advantages can lead to improved accuracy, enhanced business and strategic benefits, more reliable assumptions, improved risk mitigation, etc.

THE “GOLDEN QUESTION” Through brainstorming, feedback loops, and data review, determine what single characteristic (“golden question”) will define your target

OPERATIONAL CONSIDERATIONS • Executive & cross-functional support • Time/cost versus depth of investigation • Strategic modeling process • Cross-functional involvement throughout build • Thorough training

Executive & Cross-Functional Support If target users don’t support the model, they will resist using it. Gaining complete support can be difficult: • Resistance to change • Concern that model results will highlight current deficiencies • Lack of understanding of predictive models

Support (cont’d) I already have an established plan. I know who our target customer is. My position will be eliminated if a model is now used to select risks. My expertise must not be important to the company. The model will suggest that my current method is incorrect, which will reflect poorly on my performance/reputation. I don’t know how to explain this to a broker/agent so I don’t want to use it. I will have to take on additional work associated with new processes. My workflow will double (triple). We’ve always done it this way, and it’s worked. I don’t see a reason to change anything. I found one outlier so the model must be wrong.

Time/Cost vs. Depth of Investigation The process of building and implementing a model can typically be quite lengthy – longer than most expect OR Remember that a simple model does not necessarily indicate a simple study! • Simpler Study (3-12 months) • Results more conservative • Perhaps internal data only • Generous binning • Limited interactions • May be appropriate if goal is a general sense of direction • More thorough investigation • Additional time • Additional development cost • Possible greater payoff through enhanced segmentation and data exploration

Strategic Modeling Process TARGET PREDICTION/USE • Ensure target is appropriate for the intended use • While many ideas are interesting, you may wish to focus on those which are actionable STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACT • The most statistically significant model may not be the most impactful • Consider ease of implementation, repeatability, updates • Identify when “less is more”! FLEXIBILITY • Allow for unexpected insights which could lead to unanticipated changes in business strategy or process • Sometimes the insights gained from the journey will prove more important than the planned goal

Cross-Functional Involvement Data, product & IT experts, legal advisors, and model users must remain engaged throughout the model build

Thorough Training The model isn’t done when it’s done. Who will provide the training? Who is most appropriate to provide training? Modeling team General training team Functional experts Consulting team* Other No clear answer – but this must be thoughtfully considered and appropriately executed to reap the full benefits of the model which was built *Consider what information may be shared (non-proprietary)

Discussion/Q&A Remember… Modeling is a complete business strategy NOT just a mathematical process So how will YOU use predictive modeling to improve your business?

Christine Hofbeck, FSA, MAAA • Centroid Analytics, LLC • christine.hofbeck@centroidanalytics.com • 908.884-4103 (c) • 908.574-5351 (w) • www.centroidanalytics.com

The Business of Predictive Modeling