Choosing and validating new kits and equipment

1. Choosing and validating new kits and equipment Keith Perry, Unit HeadMicrobiological Diagnostics Assessment Service [MiDAS] Evaluations & Standards Laboratory BSMT meeting, Management in Microbiology LaboratoriesNovember 2005

2. Choosing & validating kits & equipment

3. Choosing & validating kits & equipment Today’s presentation • Background to MiDAS • Costs / Procurement • CE Marking • Evaluations • Validations • Issues common to Evaluations & Validations • Practical considerations

4. Evaluations & Standards Laboratory [ESL] Evaluations Unit[MiDAS] Quality Control Reagents Unit Standards Unit Quality SystemUnit Evaluate kits & equipment and write reports on findings Write and co-ordinate the consultation and distribution of standard methods and clinical testing algorithms Provide laboratories with virology /serology IQC reagents Advise on quality systems and audit MiDAS

5. MiDAS Associated organisations / contracts NationalBloodService Centre for Evidence-basedPurchasing MiDAS[ESL] Input into NBS Kit Evaluation Group committee Evaluation Centrefor Microbiological IVDDs Commercial contracts WorldHealthOrganisation WHO Collaborating Centre forDiagnostic and Laboratory Support

6. www.hpa-midas.org.uk

7. Procurement • Recent formation of Centre for Evidence-based Purchasing (CEP-PASA) • Focus on regional procurement hubs • Improved support for clinical networks by increasing their involvement in purchasing • Joining up device performance data with purchasing cycles • Strategic role of procurement in supporting adoption of new technology/ideas

8. www.pasa.nhs.uk/evaluation

9. CE Marking Common Technical Specifications IVDD Directive – 98/79/EC

10. CE Marking CE Marking does not cover:- • Ease of Use • Monitoring of internal QCs • External QA • Use of combinations ie kits/automated platforms • Comparative data • Evaluation and validation

11. Evaluations A generic term for any study that measures the performancecapabilities of an assay. • To undertake comparative evaluations of devices intended for the in-vitro diagnosis and management of microbiological infection • To focus on device performance • To be independent and unbiased • To undertake evaluations in an accurate, efficient and timely manner • To provide informative reports While the main focus of MiDAS is evaluation of commercial devices, our processes can also contribute to high quality evaluation of innovative and in-house assays. Aims

12. Evaluations Why do we need evaluations? • Best practice • Relevant to important public health concerns • Standardisation of methods • Managed introduction of new technology • Encourages development

13. Evaluations

14. Timing of detection of primary HIV following seroconversion Murex HIV 1/2 VK84/85 Vironostika HIV Uni-form II Ag/Ab Vitros ECi anti-HIV 1/2 Murex HIV Ag/Ab combination Ortho Ab-capt. ELISA BiotestHIV 1/2 recombinant Access HIV 1/2 NEW AxSYM HIV Ag/Ab Combo Innotest HIV-1/-2 GENSCREEN PLUS HIV Ag-Ab Enzygnost HIV 1/2 plus Biotest Anti-HIV TETRA ELISA Biotest Anti-HIV TETRA ELISA 5 0 10 15 20 days Murex HIV 1.2.0 GE94/95 Murex ICE HIV 1+2 VIDAS HIV DUO *Earliest anti-HIV detection Earliest HIV detection Abbott 3rd gen Plus Clonesystems Detect-HIV v1 Enzygnost HIV Integral Vironostika HIV Uniform II plus O IMx HIV1/2 III plus Wellcozyme Anti-HIV Pasteur Genscreen Version 2 AxSYM HIV 1/2 gO = combined antigen-antibody = immunometric = antiglobulin / indirect = Class specific antibody capture Evaluations Choosing HIV kits

15. Evaluations Choosing HIV kits Combined scores for commercial seroconversion panels

16. Evaluations Choosing HIV kits Sensitivity Specificity

17. Evaluations Good evaluation practice

18. Evaluations Defined procedures Prioritisation system Literature Review 16 stages 16 stages Preparation Setup 11 stages 10 stages Technical Assessment 13 stages 13 stages Report 13 stages 13 stages Post publication 8 stages 8 stages Customer Service & feedback

19. Evaluations Collaborations

20. Evaluations Benefits of collaborations • To incorporate latest developments and test algorithms into evaluation design • To maximise mutual benefit • To avoid conflicts in output • To bring together information on all evaluations undertaken • To work to publish results in a timely manner • To use mechanisms already in place and to avoid compromising good evaluation practice

21. Evaluations Collaborations: Chlamydia trachomatis NAATs

22. Extent of NHS use Evaluations Prioritisation – CEP ‘Hopper’ system Anyone can submit proposals Adoption/purchase Evaluation Evaluation project proposals Time Prioritisation Board • Chair • Head of CEP • Business Planning Manager • Head of Commissioning and Delivery • Technical Advisor • PASA / Hubs (2 reps) • NICE (1 rep) • NIIII (1 rep) • HTA (1 rep) • NPSA (1 rep) • MHRA (1 rep) ? Evaluations Special Interest Group Priorities to be aligned to customer requirements& peer-organisations

23. Evaluation priority survey (pilot) – Serology kits Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs Evaluations Prioritisation

24. Evaluations Evaluation priority survey (pilot) - Nucleic Acid Tests Prioritisation Chart excludes ongoing evaluations ie HIV, HBV, HCV, syphilis (total), chlamydia NAATs

25. Validation:with special thanks to Dr. Ian Sharp & Sally-Ann Finn Definition of validation • Validation is the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled (ISO 17025:2005) Validation is theevaluation of a processto determine its fitness for a particular use.

26. Validation Evaluation of the process • It is not the kit or reagent in isolation that is being validated but the whole process that it is being used in to produce the correct result • You are validating your ability to achieve acceptable results with the assay in question

27. Validation

28. Validation Why do we need to validate assays? • Good laboratory practice • Protection from litigation • CPA requirement • F 1.2 Examination procedures shall be validated for their intended use prior to introduction, and the methods used and results obtained, recorded.

29. Validation What do we need to validate? • Commercial assays • Can use the term commissioning • In house assays • Modified commercial assays • Changes to assays • Equipment-assay combinations

30. Validation Commercial assays • Just because they are CE marked does not mean that they do not need to be validated (commissioned) to ensure that they are fit for purpose • Most assays are CE marked following self-declaration by the manufacturer • May be certified to ISO 13485

31. VALIDATIONFILE Validation What’s needed? Project team Project lead Project manager Molecular skillsSerology skills Statistical skills Validation plan

32. Validation Validation file

33. Issues common to Evaluations & Validations Challenging specimen panel - Numbers

34. Issues common to Evaluations & Validations Challenging specimen panel - Numbers Confidence intervals should always be quoted

35. Carley, Dosman S, Jones SR, Harrison M (2005): Simple nomograms to calculatesample size in diagnostic studiesEmerg Med J 2005; 22: 180-181 Issues common to Evaluations & Validations Challenging specimen panel - Numbers

36. Issues common to Evaluations & Validations Challenging specimen panel – Minimising bias • Compare results with a suitable reference standard / algorithm • Avoid including specimens pre-screened by kits that are part of the evaluation • Avoid including specimens known to share false reactivities with particular groups of kits • Be aware of discontinued use of challenging specimens within a panel (eg due to low volume) which may apparently enhance sensitivity/specificity calculations of kits tested at a later date.

37. Issues common to Evaluations & Validations Challenging specimen panel – Representative types • Include specimens with High, Medium and Low reactivity, and unreactive specimens

38. Issues common to Evaluations & Validations Challenging specimen panel – in practice depends on: • Specimen availability • Specimen volume • Number of kits to evaluate • Specimen quality (eg freeze/thaw cycles) • Ethical issues • Access to diagnostic specimens • Funds eg commercial seroconversion panels

39. Practical Considerations • Technical expertise required to perform the assay • Training • Test throughput • Reliability of the supplier • Service provision & continuity • Requirement for specialised equipment • Maintenance contracts • Laboratory space

40. Challenges • Responding to the need for a wide range of microbiological device evaluations • Joining device performance with procurement • Earlier assessment and usage of innovative technologies • Improved access to available evaluation resultseg National Evaluations Register • Improved sharing of information

41. Acknowledgments Evaluations & Standards Laboratory:Ian SharpSally-Ann FinnRuhi SiddiquiJoe VinciniValerie Bevan MiDAS-ESL:Katrina BarlowMichelle ColeJohanna CurtisLaura DeanGalit GonenFu Li

42. Can You Help? Acquiring specimens