Model Monitor: Evaluating and Choosing Robust Models for Changing Conditions

Troy Raeder Nitesh V. Chawla University of Notre Dame Model Monitor: Evaluating and Choosing Robust Models for Changing Conditions

Meet Bob

Data Training Testing

Hellinger Distance KS Test Data Training Testing Distribution Changing?

Hellinger Distance KS Test P-Value Data Training Testing Distribution Changing?

Hellinger Distance KS Test P-Value New Test Distribution Shift Testing Model New Predictions Predictions KW Test Distribution Changing? Model Vulnerable?

KS Test Hellinger Distance Model Data P-Value Predictions New Test Distribution Shift Testing Rank Models Model New Predictions Predictions Friedman Test Bonferroni Dunn Test KW Test Distribution Changing? Best Model? Model Vulnerable?

Thank You • T. Raeder and N.V. Chawla “Model Monitor (M^2): Evaluating, Comparing, and Monitoring Models” http://www.cse.nd.edu/Reports/2008/TR-2008-17.pdf • D.A. Cieslak and N.V. Chawla: “Detecting Fracture Points in Classifier Performance”, ICDM 2007, • D.A. Cieslak and N.V. Chawla: “A Framework for Monitoring Classifiers Performance: When and Why Failure Occurs”, KAIS 2008 • http://www.nd.edu/~dial/ • traeder@nd.edu

Model Monitor: Evaluating and Choosing Robust Models for Changing Conditions