Adventures in Segmentation Using Applied Data Mining to add Business Value

1. Adventures in SegmentationUsing Applied Data Mining to add Business Value Drew Minkin

2. Agenda • The Value Add of Data Mining • Segmentation 101 • Segmentation Tools in Analysis Services • Methodology for Segmentation Analysis • Building Confidence in your Model

3. The Value Add of Data Mining

4. Value Add - What is Data Mining? • Statistics for the Computer Age • Evolution, not revolution with traditional statistics • Statistics enriched with brute-force capabilities of modern computing • Associated with industrial-sized data sets

5. Data Mining OLAP Reports (Ad hoc) Reports (Static) Value Add - Data Mining in the BI Spectrum Business Knowledge SQL-Server 2008 Relative Business Value Easy Difficult

6. Value Add – Data Mining and Democracy • VoterVault • From Mid-1990s • Massive get-out-the-vote drive for those expected to vote Republican • Demzilla • Names typically have 200 to 400 information items

7. Value Add – The Promise of Data Mining • “The quiet statisticians have changed our world; not by discovering new facts or technical developments, but by changing the ways that we reason, experiment and form our opinions.” -- Ian Hacking

8. Value Add – Spheres of Influence

9. Value Add – Operational Benefits • Improved efficiency • Inventory management • Risk management

10. Value Add – Strategic Benefits • The Bottom Line • Increased agility • Brand building • Differentiate message • “Relationship” building

11. Value Add – Tactical Benefits • Reduction of costs • Transactional leakage • Outlier analysis

12. Value Add - Customer Attrition Analysis • Identify a group of customers who are expected to attrite • Conduct marketing campaigns to change the behavior in the desired direction • change their behavior, reduce the attrition rate.

13. Value Add - Target Result • Slow attriters: Customers who slowly pay down their outstanding balance until they become inactive. • Fast attriters: Customers who quickly pay down their balance and either lapse it or close it via phone call or write in.

14. Value Add - Sample Applications • Credit models • Retention models • Elasticity models • Cross-sell models • Lifetime Value models • Agent/agency monitoring • Target marketing • Fraud detection

15. Segmentation 101

16. Segmentation – Machine Learning • Unsupervised learning • Associations and patterns • many entities • target information • Market basket analysis (“diapers and beer”) • Supervised learning • Predict the value • target variable • well-defined predictive variables • Credit / non-credit scoring engines

17. Segmentation –Sample Data Sources • Data Warehouse: Credit Card Data Warehouse containing about 200 product specific fields • Third Party Data : A set of account related demographic and credit bureau information • Segmentation files :Set of account related segmentation values based on our client's segmentation scheme which combines Risk, Profitability and External potential • Payment Database :Database that stores all checks processed. The database can categorize source of checks

18. Methodology for Segmentation Analysis

19. Methodology–Distribution of Effort

20. Methodology – Segmentation Lifecycle

21. Methodology – Acquiring Raw Data • Research/Evaluate possible data sources • Availability • Hit rate • Implementability • Cost-effectiveness • Extract/purchase data • Check data for quality (QA) • At this stage, data is still in a “raw” form • Often start with voluminous transactional data • Much of the data mining process is “messy”

22. Methodology – Goals of Refinement • Reflects data changes over time. • Recognizes and removes statistically insignificant fields • Defines and introduces the "target" field • Allows for second stage preprocessing and statistical analysis.

23. Methodology - Scoring Engines • Scoring engine • Formula that classifies or separates policies (or risks, accounts, agents…) into • profitable vs. unprofitable • Retaining vs. non-retaining… • (Non-)Linear equation f() of several predictive variables • Produces continuous range of scores score = f(X1, X2, …, XN)

24. Methodology – Deployed Model Data To Predict Training Data Mining Model Mining Model Mining Model DB data Client data Application log “Just one row” New Entry New Txion DM Engine DM Engine Predicted Data

25. Methodology - Testing • Randomly divide data into 3 pieces • Training data • Test data • Validation data • Use Training data to fit models • Score the Test data to create a lift curve • Perform the train/test steps iteratively until you have a model you’re happy with • During this iterative phase, validation data is set aside in a “lock box” • Score the Validation data and produce a lift curve • Unbiased estimate of future performance

26. Methodology - Multivariate Analysis • Examine correlations among the variables • Weed out redundant, weak, poorly distributed variables • Model design • Build candidate models • Regression/GLM • Decision Trees/MARS • Neural Networks • Select final model

27. Segmentation Tools in Analysis Services

28. Data Mining - Algorithm Matrix Segmentation Advanced Data Exploration Classification Forecasting Association Text Analysis Estimation Association Rules Clustering Decision Trees Linear Regression Logistic Regression Naïve Bayes Neural Nets Sequence Clustering Time Series

29. Data Mining - SQL-Server Algorithms Decision Trees Clustering Time Series Neural Net Sequence Clustering Association Naïve Bayes Linear and Logistic Regression

30. Data Mining - Blueprint for Toolset • Offline and online modes • Everything you do stays on the server • Offline requires server admin privileges to deploy • Define Data Sources and Data Source Views • Define Mining Structure and Models • Train (process) the Structures • Verify accuracy • Explore and visualise • Perform predictions • Deploy for other users • Regularly update and re-validate the Model

31. Data Mining - Cross-Validation • SQL Server 2008 • X iterations of retraining and retesting the model • Results from each test statistically collated • Model deemed accurate (and perhaps reliable) when variance is low and results meet expectations

32. Data Mining - Microsoft Decision Trees • Use for: • Classification: churn and risk analysis • Regression: predict profit or income • Association analysis based on multiple predictable variable • Builds one tree for each predictable attribute • Fast

33. Data Mining - Microsoft Naïve Bayes • Use for: • Classification • Association with multiple predictable attributes • Assumes all inputs are independent • Simple classification technique based on conditional probability

34. Data Mining - Clustering • Applied to • Segmentation: Customer grouping, Mailing campaign • Also: classification and regression • Anomaly detection • Discrete and continuous • Note: • “Predict Only” attributes not used for clustering

35. Data Mining - Neural Network Output Layer Loyalty Hidden Layers Input Layer Age Education Sex Income • Applied to • Classification • Regression • Great for finding complicated relationship among attributes • Difficult to interpret results • Gradient Descent method

36. Data Mining - Sequence Clustering • Analysis of: • Customer behaviour • Transaction patterns • Click stream • Customer segmentation • Sequence prediction • Mix of clustering and sequence technologies • Groups individuals based on their profiles including sequence data

37. Data Mining - What is a Sequence? • To discover the most likely beginning, paths, and ends of a customer’s journey through our domain consider using: • Association Rules • Sequence Clustering

38. Data Mining - Sequence Data

39. Data Mining – Minor Introduction to DMX • Your “if” statement will test the value returned from a prediction – typically, predicted probability or outcome • Steps: • Build a case (set of attributes) representing the transaction you are processing at the moment • E.g. Shopping basket of a customer plus their shipping info • Execute a “SELECT ... PREDICTION JOIN” on the pre-loaded mining model • Read returned attributes, especially case probability for a some outcome • E.g. Probability > 50% that “TransactionOutcome=ShippingDeliveryFailure” • Your application has just made an intelligent decision! • Remember to refresh and retest the model regularly – daily?

40. Data Mining – Detailed Workflow

41. Data Mining – Detailed Mining Model

42. Data Mining – Detailed Mining Model

43. Data Mining – Detailed Mining Model

44. Building Confidence in your Segmentation

45. Building Confidence - Model Design • Which target variable to use? • Frequency & severity • Loss Ratio, other profitability measures • Binary targets: defection, cross-sell • …etc • How to prepare the target variable? • Period - 1-year or Multi-year? • Losses evaluated @? • Cap large losses? • Cat losses? • How / whether to re-rate, adjust premium? • What counts as a “retaining” policy? • …etc

46. Building Confidence - Improving Models • Approaches • Change the algorithm • Change model parameters • Change inputs/outputs to avoid bad correlations • Clean the data set • Perhaps there are no good patterns in data • Verify statistics (Data Explorer)

47. Building Confidence – Alternate Methods • Capping • Outliers reduced in influence and to produce better estimates. • Binning • Small and insignificant levels of character variables are regrouped. • Box-Cox Transformations • These transformations are commonly included, specially, the square root and logarithm. • Johnson Transformations • Performed on numeric variables to make them more ‘normal’. • Weight of Evidence • Created for character variables and binned numeric variables.

48. Building Confidence - Confusion Matrix 1241 correct predictions (516 + 725) . 35 incorrect predictions (25 + 10). The model scored 1276 cases (1241+35). The error rate is 35/1276 = 0.0274. The accuracy rate is 1241/1276 = 0.9725.

49. Building Confidence – Warning Signs • “All models are wrong, but some are useful." • George Box

50. Building Confidence –Li’s Revenge • Extrapolation • Applying models from unrelated disciplines • Equality • The real world contains a surprising amount of uncertainty, fuzziness, and precariousness. • Copula • Binding probabilities can mask errors • Distribution functions • Small miscalculations can make coincidences look like certainties • Gamma • Human behavior difficult to quantify as a linear parameter