Interlaboratory Tests

1. Interlaboratory Tests Michael Koch In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

2. What are interlaboratory tests? • Randomly selected sub-samples from a source of material are distributed simultaneously to participating laboratories for concurrent testing In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

3. method validation reference material characterization proficiency testing Types of interlaboratory tests In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

4. Interlaboratory tests for the validation of a method • objective: best possible characterization of the method • laboratories have to use exactly the same method • assistance should be given to assure this In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

5. Interlaboratory tests for characte-rization of a reference material • concentration of the analyte in the material must be analysed by experienced laboratories • less experienced laboratories should not be allowed to participate • objective: best possible estimation of the “true value” of the concentration In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

6. Interlaboratory tests for proficiency testing of laboratories • objective: to get an indication of the performance of an individual laboratory or a group of laboratories as a whole • laboratories should work under routine conditions • help for the laboratory to improve its quality • can be used by customers or regulatory bodies for the selection of qualified laboratories In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

7. Objectives of proficiency tests • basic concern is accuracy • inaccuracy contains systematic and random effects • laboratory can determine, whether imprecision or bias is the reason for its inaccuracy In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

8. Motivation for the laboratories • to uncover errors that couldn’t be found with internal quality control • use as certificate for competence in this testing field for clients, authorities and accreditation bodies In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

9. Limitations • Interlaboratory tests are always retrospective • organisation, distribution of samples, analyses, evaluation take time • it is dangerous to rely only on interlaboratory tests • Proficiency tests cover only a small fraction of the often wide variety of analyses • Proficiency tests do not reflect routine analyses In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

10. Standards and guidelines for proficiency testing - I • ISO Guide 43: Proficiency testing by interlaboratory comparisons • Part 1: Development and operation of proficiency testing schemes. • Part 2: Selection and use of proficiency testing schemes by laboratory accreditation bodies. • IUPAC, ISO, AOAC (1991): The International Harmonized Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories. In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

11. Standards and guidelines for proficiency testing - II • International laboratory accreditation cooperation (ILAC): Guidelines for the requirements for the competence of providers of proficiency testing schemes. • Draft ISO 13528: Statistical Methods for the Use in Proficiency Testing by Interlaboratory comparisons. In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

12. Demands on the provider -personnel • special organizational capabilities • technical experts for the analysis • statisticians • all staff have to be competent for the work it is responsible for In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

13. Demands on the provider –Planning - I • The interlaboratory test should be carefully prepared. • The planning must be documented before the start of the test In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

14. Demands on the provider –Planning - II • The plan should typically include: • name and address of the PT provider • name and address of the coordinator and other personnel • nature and purpose of the PT scheme • procedure for the manner in which the participants are selected or criteria which have to met before participation is allowed • name and address of the laboratory performing the scheme (e.g. sampling, sampling processing, homogeneity testing and assigning values) and the number of expected participants. In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

15. Demands on the provider –Planning - III • Planning content (contd.): • nature of the test items and of the tests selected • description of the manner in which the test items are obtained, processed, checked and transported. • description of the information that is supplied to participants and the time schedule for the various phases. • information on methods or procedures which participants may need to use to perform the tests or measurements. In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

16. Demands on the provider –Planning - IV • Planning content (contd.): • outline of the statistical analysis to be used. • description of how the assigned value is determined. • description of the data or information to be returned to participants. • basis techniques and methods used for evaluation • description of the extent to which the test results are to made public. In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

17. Demands on the provider –Data-processing equipment • Equipment should be adequate for • data processing • statistical analysis • to provide timely and valid results • Software must be • verified and • backed up In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

18. Demands on the provider – Test item preparation and management - I • For the selection of the test item all characteristics that could affect the integrity of the interlaboratory comparison should be considered • homogeneity • stability • possible changes during transport • effects of ambient conditions (e.g. temperature) In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

19. Demands on the provider – Test item preparation and management - II • samples used in the proficiency test should be similar to the samples that are routinely analysed in the laboratories • sample amount • surplus of sample can be used as reference material • surplus can be used to make excessive effort on the analyses In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

20. Demands on the provider -Homogeneity - I • The PT provider has to ensure that every laboratory will receive samples that do not differ significantly in the parameters to be measured • documented procedure for establishing this homogeneity • degree of homogeneity  evaluation of the laboratories results must not be significantly affected • any variation between the portions must be negligible in relation to the expected variations between the participants In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

21. Demands on the provider -Homogeneity - II • true solutions are homogeneous at a molecular level • for solid samples  special care on the homogenisation • a formal homogeneity check is described in the „International harmonized protocol...“ In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

22. Demands on the provider -Stability - I • test material must be sufficiently stable • under the conditions of storage and distribution to the participants • for the time period from producing the samples until the analyses in the participant’s laboratory • this stability has to be tested by the PT provider In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

23. Demands on the provider -Stability - II • Analysing a part of the samples after the estimated time necessary for the distribution • differences in the results may be due to instability or to between-batch variability in the organiser’s laboratory • information may be derived from the organiser’s prior experience or obtained from technical literature • accelerated stability testing by worsening the ambient conditions for the samples In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

24. Demands on the provider -Stability - III • the organiser has to ensure that the changes due to instability do not significantly affect the evaluation of the laboratories’ performance In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

25. Choice of analytical method • Normally the laboratory should use its routine method • the choice might be limited by e.g. legal regulations • organiser should ask for details • to conduct a method specific evaluation • to give comments on the methods used In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

26. Method specific evaluation In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

27. Determination of the assigned value • one of the most critical features of a proficiency test • inappropriate value will drastically reduce the value of the scheme • the same problem as in the certification of a reference material • but the organiser of a proficiency test cannot expend the same amount of effort In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

28. Assigned value –certified reference material • ideal test material for a proficiency test • disadvantages • high costs • limited availability • in the necessary quantity • and concentration range • CRM‘s have to be stable for months and PT often deals with more or less instable samples (foodstuffs, biomedical, environmental samples) In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

29. Assigned value –Consensus of „Expert Laboratories” - I • mean of analysis by expert laboratories • with high precision reference methods and traceable materials for calibration • if different physico-chemical methods are used and the same results are obtained, it is more probable that the value is near to the „true“ value In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

30. Assigned value –Consensus of „Expert Laboratories” - II • disadvantages • very much effort to ensure the accuracy of the reference measurements • „nobody is perfect“ • there might be doubts among the participants if the result of the expert laboratories deviates from the mean of the participants In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

31. Assigned value – Formulated or “Synthetic” test materials - I • materials, spiked with the analyte to a known extent • can be made with extremely accurate amounts by gravimetric or volumetric methods • If material does not contain significant amounts of the analyte • assigned value directly from added amount • If material contains analyte, this amount has to be characterized very well. • method to calculate this content from proficiency test was recently developed by the author In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

32. Assigned value – Formulated or “Synthetic” test materials - II • disadvantages • difficult to achieve sufficient homogeneity, especially with solid materials • analyte might be bound in a different chemical form • Especially in solid materials the originally contained analyte might be bound more strongly to the matrix In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

33. Assigned value –Consensus of participants - I • easiest and cheapest way used very often • If method for analysis is easy and straightforward  good estimate of „true“ value • If a „convention method“ (an empirically defined method) is used, the consensus value is the only possibility In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

34. Assigned value –Consensus of participants - II • disadvantages • Consensus value might be seriously biased (e.g. analyses of highly volatile substances) • there might be no consensus at all • e.g. if two analytical methods are used, where one is biased • these circumstances are not uncommon in trace analysis • care should be taken to decide whether a consensus value really is good choice In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

35. Methods to calculate consensus value – Arithmetic mean • requires an outlier test • but these tests are often not very satisfactory, especially if many outliers are present • outlier tests assume normal distribution which is normally not true in trace analysis In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

36. Methods to calculate consensus value – Median • not affected by outlying data • makesnot full use of the information content of the data • if the distribution is skewed, the median is biased In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

37. Methods to calculate consensus value – Robust mean • „trimmed“ data; a certain part of the data on both tails of the data set is excluded prior to the calculation of the mean • e.g. mean of interquartile range • mean of data between the first and the third quartile of the data set • or Huber statistics In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

38. Methods to calculate consensus value – Robust mean – Huber statistics • Iterative process • define initial value for m as median of all data • all data outside m±1.5*STD are set to m+1.5*STD or m-1.5*STD • new value for m is calculated as arithmetic mean of the new data • repeat until there are no changes In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

39. Performance scoring • assigned value is the target • for the assessment of laboratories a accepted range is necessary • prescribed range originating from the demands put on the analysis (fitness for purpose) • calculated from the standard deviation of the data set In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

40. Performance scoring –Tolerance range from STD • normally distributed set of data • 95,5% of the values inside a range of ±2σ • 99,7% of the values inside a range of ±3σ • on a confidence level of 95,5 % all accurate data are inside µ±2σ In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

41. Performance scoring – Z-score • the deviation from the assigned value in standard deviation units • the standard deviation is calculated after exclusion of outlier or with robust statistics In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

42. Performance scoring – Classification of the Internat. Harmonized Protocol • |Z-score|2 - satisfactory • 2<|Z-score|3 - questionable • |Z-score|>3 - unsatisfactory • Z-scores are common practise in the assessment of laboratory results In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

43. Z-score - diagramm In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

44. Statistical distribution • Data near to the limit of determination are not normal distributed • otherwise there should be negative values with a finite probability • tolerance limits should be asymmetrical (more narrow below the assigned value, more wide above it) In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

45. with g = quality limit for Z and k1, k2 =correction factors  = rel. standard deviation  = distribution function of standard normal distribution 1- = confidence level (here: 0,955) Solution approaches for assymetrical tolerance limits • logarithmic normal distribution • take the logarithm of the values prior to statistical calculations • Modification of Z-scores In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

46. Laboratory assessment • by combination of single value assessment • involves danger of misinterpretation • a laboratory can measure one parameter permanently wrong, but nevertheless is positively assessed In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

47. Combined assessment according to Intern. Harmon. Protocol... - RSZ • RSZ (rescaled sum of z-scores) • RSZ = z/√m with m = number of scores • same scale as z-score • negative assessment, if all values are within the tolerance but a little biased in the same direction • errors with opposite sign cancel each other out In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

48. Combined assessment according to Intern. Harmon. Protocol... - SSZ • SSZ (sum of squared z-scores) • different scale, because 2-distributed • doesn‘t consider the sign of z-scores In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

49. Combination of single values assessments • just counting positive and negative assessments of all values • the absolute value of the z-score is not considered • e.g. assessment in the proficiency tests of german water authorities • 80 % of the values – |Zu-score|2 • 80 % of the parameters successful In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching

50. Test scheme reports • should be distributed to the laboratories as soon as possible • normally not later than 1 month after deadline for the analytical results • laboratories need quick feedback for corrective actions • laboratories should be identified in the report by test specific codes to maintain confidentiality In:Wenclawiak, Koch, Hadjicostas (eds.) Quality Assurance in Analytical Chemistry – Training and Teaching