A note on precision of qualitative data

A note on precision of qualitative data Tomomichi Suzuki, Tokyo University of Science szk@rs.tus.ac.jp Yusuke Tsutsumi, Mitsubishi Tanabe Pharma Corporation Natsuki Sano, Tokyo University of Science

Introduction of myself • I focus on “Statistical Data Analysis” that will bridge the gap between theory and practice. • I am attending ISQC for the fourth time • Warsaw 2004, “Effective Dynamic Process Control of Assembly Processes” • statistical control of assembly process with time dependent noise • Beijing 2007, “A Study on Adaptive Paired Comparison Experiments” • design of experiment for paired comparison • proposal on the Swiss tournament system • Seattle 2010, “Improving Taguchi’s linear graphs for split-plot experiments” • proposed new linear graphs for expressing interaction of whole-plots and sub-plots

Outline of Today’s Talk • Introduction • Precision for Quantitative Data • Precision for Qualitative Data • Comparison • Conclusions

Introduction • ISO 5725 accuracy (trueness and precision) of measurement methods and results • Tests performed on presumably identical materials in presumably identical circumstances do not, in general, yield identical results.

Introduction • ISO 5725 accuracy (trueness and precision) of measurement methods and results • Trueness: • refers to the closeness of agreement between the arithmetic mean of a large number of test results and the true or accepted reference value. • Precision: • refers to the closeness of agreement between test results.

ISO/TC 69/SC 6 • ISO/TC 69 (Application of Statistical Methods)/SC 6 (Measurement Methods and Results)/WG1 (Accuracy of measurement methods and results) is preparing a document (TR: Technical Report) on precision of qualitative data. Now in Preliminary Work Item. • Reviewed existing methods and established methods.

ISO/TC 69/SC 7 • ISO/TC 69 (Application of Statistical Methods)/SC 7 (Six Sigma) published ISO TR 14468 “Selected illustrations of attribute agreement analysis” • This is based on kappa coefficient approach.

ISO/TC 34/SC 9 • ISO/TC 34 (Food products) /SC 9 (Microbiology) produced ISO 16140 “Microbiology of food and animal feeding stuffs — Protocol for the validation of alternative methods” in 2003. • It includes method by Langton et al. (2002) • ISO/TC 34/SC 9 is revising ISO 16140 “Microbiology of food and animal feed — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method”. • It includes method by Wilrich.

AOAC InternationalISO/TC 34/SC 16 • ISPAM (International Stakeholder Panel on Alternative Methods)produced a document on “Guidelines for Validation of Qualitative Chemistry Methods” which is based on POD model proposed by P. Wehling et al. (2011) • This is the main part of the ISO/TC 34 (Food products) /SC 16 (Horizontal methods for molecular biomarker analysis) document. “Validation Scheme for Qualitative Analytical Methods” (possible alternative title: "Performance characteristics and validation of binary measurement methods")

IMEKO/TC 21 • IMEKO (International Measurement Confederation) / TC 21 (Mathematical Tools for Measurements) hold SIG (Special Interest Group) “Precision evaluation in non-quantitative measurements”.

Purpose • Many methods are proposed for qualitative data, but their effectiveness and statistical properties are not so clear. • This paper introduces the methods to evaluate precision for qualitative data, then proposes a method using logit model. The proposed method is compared with existing methods.

Collaborative Assessment Experiment • Every laboratory measures the identical test item number of times.

Precision for Quantitative Data • Repeatability: • is the precision under repeatability conditions • where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time. • Repeatability indicates the smallest variation for a particular measurement method. • Reproducibility: • is the precision under reproducibility conditions • where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment. • Reproducibility indicates the largest variation for a particular measurement method.

Precision for Quantitative Data • Model used in ISO 5725 y = m + B + e • y is the measurement result • m is the general mean (expectation) • B is the laboratory component of bias under repeatability conditions (variance sL2) • e is the random error in every measurement under repeatability conditions. (variance se2)

Precision for Quantitative Data • Repeatability variance sr2 sr2 = se2 , or • Reproducibility variance sR2 sR2 = sL2 + sr2 = sL2 + se2 , or (1) • The estimates of repeatability variance and reproducibility variance are calculated from interlaboratory studies or collaborative assessment experiments.

Precision for Quantitative Data • Gauge R & R • Many objects are measured (there are variation in products) • Gauge Repeatability = Repeatability in ISO 5725 (sr2) • Gauge Reproducibility ≠ Reproducibility in ISO 5725 • Gauge Reproducibility = Between Laboratory Variance in ISO 5725 (sL2) sR2 = sL2 + sr2 (1)

Precision for Qualitative Data • Non-quantitative measurements • binary data, categorical data, ordinal data, etc. • In this paper, the methods to evaluate precision for binary data are considered. • Methods compared are • Accordance and concordance (Langton’s) • Attribute agreement analysis (Kappa) • van Wieringen’s method • Proposed method

Precision for Quality Data • The value of yik is either 0 (negative, non-detect, fail, etc.) or 1 (positive, detect, pass, etc.). Table 1

Qualitative Methods0. ISO Based Method ：general mean Wilrich’s model ：laboratory component of bias ISO5725 model ：random error Repeatability Variance ：Estimate of probability of detecting for lab i(i=1, 2, …, L) ：number of repetitions (measurements) Inter-laboratory Variance Component Reproducibility Variance

Qualitative Methods1. Accordance and Concordance Lab i １１ matching probability Ai １ Accordance １０：Accordance for laboratory i (i=1, 2, ..., L) ：Accordance ：number of ‘detect’ for lab i(i=1, 2, ..., L) ：number of repetitions (measurements)

Qualitative Methods1. Accordance and Concordance lab i’’ lab i lab i’ matching probability Ci ０１１１１１１１１ Concordance ０１１０１１：Accordance for laboratory i (i=1, 2, ..., L) ：Concordance for laboratory i (i=1, 2, ..., L) ：Concordance ：number of ‘detect’ for lab i(i=1, 2, ..., L) ：number of repetitions (measurements)

Qualitative Methods1. Accordance and Concordance • Relation between ISO based method and accordance, concordance ,

Qualitative Methods2. Van Wieringen et al(2008) ・binary 1: detect, pass 0: no-detect, fail ：measurement ：true value model items appraisers repetitions true probability of ‘pass’ Sensitivity where Specificity latent class model

Qualitative Methods2. Van Wieringen et al(2008) Likelihood when where Maximum Likelihood Estimate using EM algorithm Maximum Value: Likelihood when and Maximum Likelihood Estimate using EM algorithm Maximum Value:

Qualitative Methods2. Van Wieringen et al(2008) Repeatability Reproducibility

Qualitative Methods3. Attribute Agreement Analysis Fleiss’ Kappa Statistic ：Probability that results actually matched n ：number of items m ：number of appraisers ：Probability that results match by chance M ：number of categories ：Ratio of category j ：number of item icategorized asj k = 1 complete agreement k = 0 the same as chance (no correlation) k = -1 complete non-agreement k within appraisers k between appraisers

Qualitative Methods4. Proposed Method • We propose the method of estimating repeatability and reproducibility using the logit transformation. • When the number of positive results xi follows a binomial distribution with parameters n and pi, then logit transformation of pi* asymptotically follows a normal distribution as follows. (4) where and

Qualitative Methods4. Proposed Method • When we consider xi as the measurement result of laboratory i, the variances can be estimated by means of one-way ANOVA as shown below. • Repeatability Variance (6) • Reproducibility Variance (7)

Comparison of methods • Methods are compared using the same set of data in order to clarify the relation of among the methods. • Accordance and concordance, • Attribute agreement analysis (Kappa) • van Wieringen’s method • Proposed method • We compared the methods by averaging the obtained precision measures in the case for Langton's method and the proposed method. • The parameters are set based on van Wieringen's method.

Comparison of methods • Values of parameters (of figures next page) • overall p (probability of an item being conforming): 0.5 • number of items: 200 • number of raters L: 3 • number of repetitions for each rater n: 3 • probability of evaluating conforming item as pass: 0.99, 0.95, 0.90 • probability of evaluating nonconforming item as pass: 0.01, 0.05, 0.10 (those probabilities are for raters 1 to 3 respectively)

Comparison of methodsResults • Repeatability(above) and Reproducibility(below) • Strong relation among the methods. But not identical.

Comparison of methodsResults • Accordance and concordance, attribute agreement analysis and the proposed method gave very similar results. • The method proposed by van Wieringen also gave similar results but the relationship was not as strong. • The reason for giving different precision measures should be investigated.

Conclusions • The method utilizing logit transformation is proposed. • The proposed method and other existing methods are compared using the same set of data. • Accordance and concordance, attribute agreement analysis and the proposed method gave very similar results. • The method proposed by van Wieringen also gave similar results but the relationship was not as strong. • Other methods (POD models etc.) should also be compared. How to compare is the problem. • It would be expected that these findings contribute to standardization of evaluating precision of binary measurements.

References • Danila O., Steiner, S. H., and Mackay R. J. (2008). Assessing a Binary Measurement System. Journal of Quality Technology, 40, 310-318. • Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions. 2nd edition, John Wiley & Sons. • Horie K., Tsutsumi Y., Suzuki T. (2008). Calculation of Repeatability and Reproducibility for Qualitative Data. Proc. 6th ANQ Congress, 12 pages (CDROM). • ISO 5725 (1994). Accuracy (trueness and precision) of measurement methods and result – Part 1 : General principles and definitions. • ISO 5725 (1994). Accuracy (trueness and precision) of measurement methods and result – Part 2 : Basic methods for the determination of repeatability and reproducibility of a standard measurement methods. • ISO/TR 14468 (2010). Selected illustrations of attribute agreement analysis. • Langton, S.D., Chevennement, R., Nagelkerke N., and Lombard B. (2002). Analysingcollaborative trials for qualitative microbiological methods: accordance and concordance. International Journal of Food Microbiology, 79, 175-181.

References • Mandel, J. (1997). Repeatability and Reproducibility for Pass/Fail Data. Journal of Testing and Evaluation, 25, 151-153. • Van der Voet, H. and van Raamsdonk L. W. D. (2004). Estimation of accordance and concordance in inter-laboratory trials of analytical methods with qualitative results. International Journal of Food Microbiology, 95, 231-234. • Wehling, P., LaBudde, R.A., Brunelle, S. L., and Nelson, M. T. (2011). Probability of Detection (POD) as a statistical model for the validation of qualitative methods. Journal of AOAC International, 94, 335-347. • Van Wieringen, W. N., and de Mast, J. (2008). Measurement System Analysis for Binary Data. Technometrics, 50, 468-478. • Wilrich, P.-Th. (2010). The determination of precision of qualitative measurement methods by interlaboratory experiments. Accreditation and Quality Assurance, 15, 439-444.

Thank you for your attention!

A note on precision of qualitative data