Validation of Predictive Models: Acceptable Prediction Zone Method

1. Validation of Predictive Models: Acceptable Prediction Zone Method Thomas P. Oscar, Ph.D. USDA, Agricultural Research Service Microbial Food Safety Research Unit University of Maryland Eastern Shore Princess Anne, MD

2. Background Information

3. Terminology • Performance evaluation • Process of comparing observed and predicted values. • Validation • A potential outcome of performance evaluation. • Requires establishment of criteria.

4. Test Data Interpolation Extrapolation Performance Bias Accuracy Systematic Bias Criteria

5. Predictive Modeling Secondary Models Tertiary Model No Model Observed No Predicted No Observed N(t) l Model Observed l Predicted l Primary Model Primary Model mmax Model Observed mmax Predicted mmax Predicted N(t) Predicted N(t) Nmax Model Observed Nmax Predicted Nmax

6. Performance Evaluation Stage 1 Goodness-of-fit Primary/Secondary Models Verification Tertiary Models Stage 2 Interpolation All Models Stage 3 Extrapolation All Models

7. Test Data CriteriaInterpolation • Independent data. • Within the response surface. • Uniform coverage. • Collected with same methods. Incomplete and biased evaluation Model data (10 to 40C) versus Test data (25 to 40C)

8. Test Data CriteriaExtrapolation • Independent data. • Outside the response surface. • Only one variable differs. • Collected with same methods. Confounded comparison Strain A in broth versus Strain B in food

9. Acceptable Prediction Zone MethodDescription

10. Relative Error (RE) RE for  = (predicted - observed)/predicted RE for N(t), No, max and Nmax = (observed - predicted)/predicted RE < 0 are “fail-safe” RE > 0 are “fail-dangerous”

11. Performance Factor%RE = REIN/RETOTAL

12. Performance Criteria • Acceptable Predictions -0.30 < RE < 0.15 for mmax -0.60 < RE < 0.30 for l -0.80 < RE < 0.40 for N(t), No, Nmax • Acceptable Performance %RE => 70

13. Acceptable Prediction Zone MethodDemonstration

14. Model Development Design • Salmonella Typhimurium • No = 4.8 log CFU/g • Sterile cooked chicken • 10, 12, 14, 16, 20, 24, 28, 32, 36, 38, 40C • Viable counts • BHI agar • 12 per growth curve

15. Performance Evaluation DesignSecondary Models (Interpolation) • Salmonella Typhimurium • No = 4.8 log CFU/g • Sterile cooked chicken • 11, 13, 15, 18, 22, 26, 30, 34, 37, 39C • Viable counts • BHI agar • 12 per growth curve

16. Primary ModelLogistic with Delay N = No if t N = Nmax/(1+[(Nmax/No)-1]exp[-max (t-)]) if t > 

17. Primary Model PerformanceGoodness-of-fit

18. Secondary Model for No No = mean No

19. No Model Performance

20. Secondary Model for lHyperbola with Shape Factor  = [41.47/(T - 7.325)]1.44

21. l Model Performance

22. Secondary Model for mmaxModified Square Root max = 0.01885 if T 11.43 max = 0.01885 + [0.004325(T – 11.43)]1.306if T > 11.43

23. mmax Model Performance

24. Secondary Model for NmaxAsymptote Model Nmax = exp(2.348[((T – 9.64)(T – 40.74))/((T – 9.606)(T – 40.76))])

25. Nmax Model Performance

26. Predictive Modeling Secondary Models Tertiary Model No Model Observed No Predicted No Observed N(t) l Model Observed l Predicted l Primary Model Primary Model mmax Model Observed mmax Predicted mmax Predicted N(t) Predicted N(t) Nmax Model Observed Nmax Predicted Nmax

27. Tertiary Model PerformanceVerification %RE = 90.7

28. Comparison of Models Fisher’s exact test; P = 0.48, not significant.

29. Performance Evaluation DesignTertiary Model (Interpolation) • Salmonella Typhimurium • No = 4.8 log CFU/g • Sterile cooked chicken • 11, 13, 15, 18, 22, 26, 30, 34, 37, 39C • Viable counts • BHI agar • 4 per growth curve

30. Tertiary Model PerformanceInterpolation

31. Tertiary Model PerformanceInterpolation %RE = 97.5

32. Should the validated tertiary model be used to predict chicken safety? • Evaluation for extrapolation to: • other initial densities (No)  • other strains • other chicken products

33. Performance Evaluation DesignTertiary Model (Extrapolation) • Salmonella Typhimurium • No = 0.8 log CFU/g • Sterile cooked chicken • 10, 12, 14, 16, 20, 24, 28, 32, 36, 40C • Viable counts • BHI agar • 4 per growth curve

34. Tertiary ModelExtrapolation to low No

35. Tertiary Model PerformanceExtrapolation to low No %RE = 2.5

36. Conclusions • Criteria are important for evaluating performance of models. • Consensus on validation would improve the quality and use of predictive models in the food industry.