Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th , 2009

1. Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15th, 2009 CAESARComputer Assisted Evaluation – Screening And ReportingThe Cervical Screening Wales Focal Point Project

2. Acknowledgements • The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Wrexham • Bryan Rose • Helen Beer and the CSW information team • David Addington-Hall and Karen Winder for advice during the scoping phase • Drs Nick Dallimore, Sally Williams and Tom Hockey • Wilma Anderson and Luke Nottage of

3. Background • Rapidly changing programme • HPV Vaccination • New technology (HPV testing, Computer Assisted Screening!!) • Modernisation (ScHARR1 and NHS CSP2 guidance, WAG pathology modernisation initiatives3) • Option appraisal: Assessment of a seven-day turn-around for the reporting of cervical smear results. School of Health and Related Research, University of Sheffield. Jan 2006 • NHS CSP: Achieving a 14 day turnaround for results by 2010. Advice to the NHS. April 2008. • Welsh Assembly Government. Getting Results: A Strategy for Diagnostic Services in Wales, 2004. Diagnostic Services Programme: The Future Delivery of Pathology Services in Wales, 2007

4. Background • Pressures on the Laboratory Service • Staff leaving • Staff recruitment and training • Fewer cases being screened – can we maintain our skills? • Challenges around service delivery – CAS is expensive technology and if beneficial, will require a laboratory service that is structured to take full advantage of it

5. Background • What is Rapid Quality Assurance screening? • Designed to improve the quality of primary screening in the cytology lab • Can be either preview or review • Also used to detect false negative and positive cases • Is therefore a requirement in calculating individual and laboratory sensitivity and specificity

6. Why CAS? • Since the introduction of LBC, screener sensitivity has improved. Calculation of sensitivities is becoming increasingly invalid as the manual QA methods have become unfit for this purpose • Anecdotally, Rapid Preview/Review isn't particularly well liked by staff

7. Some staff perform better at QA screening than others – leads to inconsistencies which will affect the calculation of individual and laboratory sensitivities • Automated screening might therefore provide a more sensitive QA approach that is also more consistent in its application

8. Introducing the Becton Dickenson Focal Point Slide Profiler

9. Focal Point GS – An Overview • The BD FocalPoint™ slide profiler is an automated imaging system designed to assist in the primary screening of SurePath™ and conventional Papanicolaou cervical cytology slides. • Designed to detect slides with evidence of squamous carcinoma and adenocarcinoma and their usual precursor conditions. • Up to 300 individual features are evaluated by the BD FocalPoint™ by using both morphometric and densitometric algorithms. • The BD FocalPoint™ identifies up to 25% of successfully processed slides as requiring No Further Review (NFR). • When used in conjunction with a BD FocalPoint™ Guided Screener Review Station the 10 highest scoring Fields Of View (FOV) from each slide are presented to the screener for review via an automated stage fitted to a standard laboratory microscope

10. Loading FocalPoint Add barcode labelled SurePath™ slides to a slide tray (8 slides per tray) LOAD & LEAVE Load slide trays into input hopper of BD FocalPoint™ slide profiler (max 36 slide trays at any one time = 288 slides) Racks can be loaded and unloaded at any time Minimum 120 slides per run There is no maximum number of slides per run

11. BD FocalPoint™ Slide Profiler processing steps • 20x Scan (High Resolution) • Each FOV scanned twice • A high resolution image is acquired for each of the 1000 FOV’s • FOV processors - segment all objects in each acquired FOV image (segmentation = separation of meaningful objects from background) • Objects classified as single cells, groups and thick groups • A single cell, group and thick group score is given for each FOV • The FOV scores are accumulated and integrated into a final slide score between 0 and 1

12. GS Review Station BD FocalPoint™ Slide Profiler Database BD FocalPoint™ GS Imaging Workflow:

13. BD FocalPoint™ GS Imaging data transfer: FocalPoint GS GS Review Station Data transfer via Removable hard-drive or VPN connection Data transfer Server PC GS Review Station

14. Principal Hypothesis • The Focal Point GS slide profiler technology is an acceptable replacement for the currently established processes of quality assurance in primary screening

15. The Project Aims • Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles • Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

16. The Project Aims • Evaluate Focal Point GS as a means of assessing sample quality • establishing baseline parameters for assessing adequacy of samples by creating a reproducible parameter for the assessment of sample taker performance • Evaluate Focal Point GS across an integrated laboratory network • assessing sample identification, tracking and transportation processes, workload allocation patterns and logistics and user friendliness

17. The Project Aims • Monitoring screener performance using the Focal Point Slide Profiler • This technology has the capability to monitor operator/observer functions such as screener opinion and action, time taken to perform screen. • These data items - gathered using the Location Guided Screener workstation (LGS) can be compared against primary screening and/or final outcome in order to provide screener performance indicators.

18. Examine Operational Issues • Identify sample identification issues e.g. bar codes and maintaining “chain of custody” across a network • How does the technology deal with inadequate samples – does this compare with current adequacy criteria? • Does the technology compare with manual techniques in terms of operator time?

19. PROCESS

20. Process • Samples received and processed at laboratory as normal • Despatched to Medical Solutions for FP scanning • Returned with data disc for Location Guided Screen (LGS) assessment in laboratory Automated Arm) • Manually screened independently of the LGS evaluation (Manual Arm) • Results correlated before final report issued

21. The ‘Automated Arm’ • Consisted of screener evaluation of 10 fields of view (FOV) on each sample • Difficult cases were referred to a checker who also examined the 10 FOV • Findings were recorded on the proforma illustrated overleaf • Outcome was recorded on LIMS (TelePath)

22. The ‘Manual’ Arm • Slides were primary screened and QA screened as usual following standard laboratory protocols • Codes and reports were recorded on TelePath • Automated arm report codes were recorded • FPQ or FPQQ codes were recorded depending on whether or not a discrepancy was noted between the two arms • Cases were authorised in the normal manner • CSW retrieved FP/LGS data as part of weekly downloads

23. CERVICAL SCREENING WALESFOCAL POINT PROJECT - PROCESS MAP AUTOMATED ARM COURIER SAMPLE TO LAB PROCESS SLIDES TO NOTTINGHAM SLIDES ARE READ ON FOCAL POINT INDIVIDUALS CREENERS OPINION OF 10 F.O.V. HARD DRIVE + PRINT OUT Process Review data RESULTS AND SLIDES RETURNED TO LAB CODE RESULTS ON TELEPATH SLIDE WIZARD NETWORK DATA TRANSFER AVAILABLE DESIGNATED PROCEDURE CODES FULL SCREEN Via Manual Arm QC DOUBLE SCREEN/ RAPID REVIEW RESULTS TO CSW PROJECT EVALUATION MANUAL ARM

24. Data and Morphology Anomaly Checking • Automated:manual arm mismatches were highlighted by the CSW Information department as and when they arose • Any resultant slide morphology issues were addressed in team assessment sessions at Med Sol in Nottingham and resolved by: - LGS/multi-header microscope sessions involving representatives from Labs / CSW/ Med Sol

25. RESULTS

26. Total Slides Processed

27. QA beyond the Lab -Focal Point GS as a means of assessing sample quality • FP reports on the Squamous and Endocervical components detected in the sample • Potentially of use to the screening programme as an indicator of smear taker proficiency – but emotive!!! • Wales monitors TZ reporting – however generally a wide spread – between 80% - 96% reported by NW labs • The study has shown that the FP reported endocervical component detection rates in NW labs only varied between 79% and 85% • Possibly screener interpretation – CAS may provide a more uniform assessment • Further work is ongoing in this area

28. Focal Point GS performance across an integrated laboratory network • VPN data transfer means that scanned slide information is available anywhere within the combined laboratory service • Economies of scale are definite considerations when planning the introduction of this technology – possible consideration for LBC procurement • Therefore, in this operational scenario – probably more efficient to combine the sample processing and CAS on one processing site.

29. Teaching and Case review using the Focal Point GS • LGS can mark and record interesting features within a slide and direct the user to these areas on the slide on demand • With a system configured using VPN these slides can be viewed anywhere on the network • Considerable potential for teaching and MDT use when combined with a multi-head microscope/projector

30. Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles

31. > 15% QC Negative Abnormal Sorting & Ranking (Illustration) The FocalPoint™ SORTS and RANKS slides between the values of 0 and 1 : based on the likelihood of abnormality being present. 0 is negative : 1 is abnormal x < 25% Archive > 75% Review Q5 Q4 Q3 Q2 Q 1

32. Note: this category Contains PR as Well as NFR samples FP Quintile Distribution

33. FP Quintile Distribution

34. Focal Point Scant Cellularity Category (FPSC) Compared to a Final Report of Inadequate • Comparison of the inadequacy levels showed that the manual arm had a similar inadequate rate – not what was expected with Wales operating a 15000 cell threshold • Interestingly the majority of cases did not correspond between the manual and automated arms

35. Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples “NFR” – DOES IT WORK?

36. Samples Assigned for NFR Compared to Final Result

37. What was missed by NFR? • 1. Cases from all labs, NFR negative, finally reported as abnormal • 78 cases in a total of 2842 • (Picked up by 10 screener, reviewer or checker) • 2.74% of cases missed by NFR – False Negatives

38. What was missed by NFR? • In over 50 000 cases manually rapid previewed by 4 Welsh Labs during Jan – September 2009 (Picked up by 10 screener, reviewer or checker) The overall False Negative Rate was 3.5% Statistically – Significant difference from 1. NFR is therefore at least as effective as Manual Rapid Pre-screen as a QA tool for primary screening of cervical samples.

39. Overall Sensitivity • In total, the automated QC screening arm did not “identify” 12 HG cases (histologically confirmed) in a total of 134 reported Sensitivity (134-12)/134 = 91% • Compare this to a quoted range of 75-95%* HG sensitivity for conventional Rapid Preview QA methods • 202 LG cases were missed or consigned to NFR in 788 reported LG samples Sensitivity (788-202)/788 = 74.4% * See references and CSW data

40. But……….! • On review, morphological abnormalities were present in the majority of 10 FOVs in the HG cases “missed “ by the automated arm. • So potentially therefore, 91% is the minimum sensitivity rate that can be expected • This is because the FocalPoint GS presented the abnormalities to the LGS user, but not identified by the user for a number of possible reasons • This may be due to a number of reasons e.g. unfamiliarity and further analysis currently being performed on this data

41. In Summary……… • Automated QC screen appears to be no less effective than rapid pre-screen • Most (161/219) cases not identified by automated arm were borderline HPV cases – FP does not seem to recognise koilocytes! – especially with minimal nuclear abnormalities. • Despite this finding, sensitivity of the automated QC screen appears to be appreciably greater in comparison to available data on manual QC methods.

42. User experience is a large factor inthe efficiency of the screen • A high proportion of the undetected abnormalities were shown to be present in the FOV’s on review – but not identified by the screener. • Staff had only received 6-8months experience of the technology - further work is currently ongoing to see if screener performance improved with time/experience

43. Monitoring laboratory and screener performance using the Focal Point Slide Profiler

44. Individual LaboratoryProcess Review Results Process Review: Is an indicator of the number of failed slide scans in a given FP run It reflects technical quality, bar code integrity etc. Note the low value for YGC, which has been mainly attributed to tape coverslipping

45. Screening times for 10 FOV Note the remarkable similarity of the mean screening times. Note also that NFR category accounts for 22% of samples - no FOVs, therefore potential productivity increase.

46. Manual QC Screening Times(includes data entry on LIMS)

47. OBSERVATIONS

48. Observations • Thought that FP operates on TBS inadequacy threshold of 5-6k cells, therefore we hypothesised that the manual arm would identify a correspondingly higher number of inadequate samples, however…………….. • Comparison of the inadequacy levels showed that the manual arm had only a slightly higher inadequate rate – not as expected with Wales operating a 15000 cell threshold • Interestingly the majority of cases did not correspond between the manual and automated arms – possibly indicative of the fact that the FPSC report is applied to cases that exhibit other criteria, such as:

49. Observations • The slide cannot be focussed on correctly because of bubbles, too thick, too few cells in the centre of the prep. etc. • It is therefore likely that the actual number of cases reported as FPSC for reasons related to cellular adequacy are lower than first thought • Further work will be undertaken to determine threshold values of the criteria that make up the FP report “Insufficient Reference Cells” to determine if this could becompared with the manual arm report of “Inadequate due to scant cellularity”

50. Observations • Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby reducing the process review rate and decreasing the number of rejected slides. • Selecting a large coverslip size on the tape cover slipper extended the FP scan time from approx. 11 slides/min on glass coverslips to 9 slides/min using tape coverslips. Reducing the tape size reduced this to 10 slides/min* • Staff used to primary screening on a x20 objective may have initial difficulties in operating the LGS which only displays FOVs at x10 • *Note: Updated versions of the FP GS allow faster scan rates