dave nuttall network manager and scientific advisor phw staff conference oct 15 th 2009
Download
Skip this Video
Download Presentation
Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th , 2009

Loading in 2 Seconds...

play fullscreen
1 / 60

Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th , 2009 - PowerPoint PPT Presentation


  • 65 Views
  • Uploaded on

Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th , 2009. CAESAR Computer Assisted Evaluation – Screening And Reporting The Cervical Screening Wales Focal Point Project. Acknowledgements.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15 th , 2009' - petra


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
dave nuttall network manager and scientific advisor phw staff conference oct 15 th 2009
Dave Nuttall

Network Manager and Scientific Advisor

PHW Staff Conference – Oct 15th, 2009

CAESARComputer Assisted Evaluation – Screening And ReportingThe Cervical Screening Wales Focal Point Project
acknowledgements
Acknowledgements
  • The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Wrexham
  • Bryan Rose
  • Helen Beer and the CSW information team
  • David Addington-Hall and Karen Winder for advice during the scoping phase
  • Drs Nick Dallimore, Sally Williams and Tom Hockey
  • Wilma Anderson and Luke Nottage of
background
Background
  • Rapidly changing programme
    • HPV Vaccination
    • New technology (HPV testing, Computer Assisted Screening!!)
    • Modernisation (ScHARR1 and NHS CSP2 guidance, WAG pathology modernisation initiatives3)
  • Option appraisal: Assessment of a seven-day turn-around for the reporting of cervical smear results. School of Health and Related Research,

University of Sheffield. Jan 2006

  • NHS CSP: Achieving a 14 day turnaround for results by 2010. Advice to the NHS. April 2008.
  • Welsh Assembly Government. Getting Results: A Strategy for Diagnostic Services in Wales, 2004.

Diagnostic Services Programme: The Future Delivery of Pathology Services in Wales, 2007

background1
Background
  • Pressures on the Laboratory Service
    • Staff leaving
    • Staff recruitment and training
    • Fewer cases being screened – can we maintain our skills?
    • Challenges around service delivery – CAS is expensive technology and if beneficial, will require a laboratory service that is structured to take full advantage of it
background2
Background
  • What is Rapid Quality Assurance screening?
  • Designed to improve the quality of primary screening in the cytology lab
  • Can be either preview or review
  • Also used to detect false negative and positive cases
  • Is therefore a requirement in calculating individual and laboratory sensitivity and specificity
why cas
Why CAS?
  • Since the introduction of LBC, screener sensitivity has improved. Calculation of sensitivities is becoming increasingly invalid as the manual QA methods have become unfit for this purpose
  • Anecdotally, Rapid Preview/Review isn\'t particularly well liked by staff
slide7
Some staff perform better at QA screening than others – leads to inconsistencies which will affect the calculation of individual and laboratory sensitivities
  • Automated screening might therefore provide a more sensitive QA approach that is also more consistent in its application
focal point gs an overview
Focal Point GS – An Overview
  • The BD FocalPoint™ slide profiler is an automated imaging system designed to assist in the primary screening of SurePath™ and conventional Papanicolaou cervical cytology slides.
  • Designed to detect slides with evidence of squamous carcinoma and adenocarcinoma and their usual precursor conditions.
  • Up to 300 individual features are evaluated by the BD FocalPoint™ by using both morphometric and densitometric algorithms.
  • The BD FocalPoint™ identifies up to 25% of successfully processed slides as requiring No Further Review (NFR).
  • When used in conjunction with a BD FocalPoint™ Guided Screener Review Station the 10 highest scoring Fields Of View (FOV) from each slide are presented to the screener for review via an automated stage fitted to a standard laboratory microscope
loading focalpoint
Loading FocalPoint

Add barcode labelled SurePath™ slides to a slide tray (8 slides per tray)

LOAD

&

LEAVE

Load slide trays into input hopper of BD FocalPoint™ slide profiler (max 36 slide trays at any one time = 288 slides)

Racks can be loaded and unloaded at any time

Minimum 120 slides per run

There is no maximum number of slides per run

bd focalpoint slide profiler processing steps
BD FocalPoint™ Slide Profiler processing steps
  • 20x Scan (High Resolution)
    • Each FOV scanned twice
    • A high resolution image is acquired for each of the 1000 FOV’s
  • FOV processors - segment all objects in each acquired FOV image (segmentation = separation of meaningful objects from background)
    • Objects classified as single cells, groups and thick groups
    • A single cell, group and thick group score is given for each FOV
  • The FOV scores are accumulated and integrated into a final slide score between 0 and 1
bd focalpoint gs imaging workflow

GS Review Station

BD FocalPoint™

Slide Profiler

Database

BD FocalPoint™ GS Imaging Workflow:
bd focalpoint gs imaging data transfer
BD FocalPoint™ GS Imaging data transfer:

FocalPoint GS

GS Review Station

Data transfer via

Removable hard-drive

or VPN connection

Data transfer

Server PC

GS Review Station

principal hypothesis
Principal Hypothesis
  • The Focal Point GS slide profiler technology is an acceptable replacement for the currently established processes of quality assurance in primary screening
the project aims
The Project Aims
  • Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles
  • Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples
the project aims1
The Project Aims
  • Evaluate Focal Point GS as a means of assessing sample quality
    • establishing baseline parameters for assessing adequacy of samples by creating a reproducible parameter for the assessment of sample taker performance
  • Evaluate Focal Point GS across an integrated laboratory network
    • assessing sample identification, tracking and transportation processes, workload allocation patterns and logistics and user friendliness
the project aims2
The Project Aims
  • Monitoring screener performance using the Focal Point Slide Profiler
    • This technology has the capability to monitor operator/observer functions such as screener opinion and action, time taken to perform screen.
    • These data items - gathered using the Location Guided Screener workstation (LGS) can be compared against primary screening and/or final outcome in order to provide screener performance indicators.
examine operational issues
Examine Operational Issues
  • Identify sample identification issues e.g. bar codes and maintaining “chain of custody” across a network
  • How does the technology deal with inadequate samples – does this compare with current adequacy criteria?
  • Does the technology compare with manual techniques in terms of operator time?
process1
Process
  • Samples received and processed at laboratory as normal
  • Despatched to Medical Solutions for FP scanning
  • Returned with data disc for Location Guided Screen (LGS) assessment in laboratory Automated Arm)
  • Manually screened independently of the LGS evaluation (Manual Arm)
  • Results correlated before final report issued
the automated arm
The ‘Automated Arm’
  • Consisted of screener evaluation of 10 fields of view (FOV) on each sample
  • Difficult cases were referred to a checker who also examined the 10 FOV
  • Findings were recorded on the proforma illustrated overleaf
  • Outcome was recorded on LIMS (TelePath)
the manual arm
The ‘Manual’ Arm
  • Slides were primary screened and QA screened as usual following standard laboratory protocols
  • Codes and reports were recorded on TelePath
  • Automated arm report codes were recorded
    • FPQ or FPQQ codes were recorded depending on whether or not a discrepancy was noted between the two arms
  • Cases were authorised in the normal manner
  • CSW retrieved FP/LGS data as part of weekly downloads
slide23

CERVICAL SCREENING WALESFOCAL POINT PROJECT - PROCESS MAP

AUTOMATED ARM

COURIER

SAMPLE

TO LAB

PROCESS

SLIDES TO

NOTTINGHAM

SLIDES ARE READ

ON FOCAL POINT

INDIVIDUALS CREENERS

OPINION OF 10 F.O.V.

HARD DRIVE +

PRINT OUT

Process Review data

RESULTS AND

SLIDES RETURNED

TO LAB

CODE RESULTS

ON TELEPATH

SLIDE WIZARD

NETWORK

DATA TRANSFER

AVAILABLE

DESIGNATED

PROCEDURE CODES

FULL SCREEN

Via

Manual Arm

QC

DOUBLE SCREEN/

RAPID REVIEW

RESULTS TO CSW

PROJECT

EVALUATION

MANUAL ARM

data and morphology anomaly checking
Data and Morphology Anomaly Checking
  • Automated:manual arm mismatches were highlighted by the CSW Information department as and when they arose
  • Any resultant slide morphology issues were addressed in team assessment sessions at Med Sol in Nottingham and resolved by:

- LGS/multi-header microscope sessions involving representatives from Labs / CSW/ Med Sol

qa beyond the lab focal point gs as a means of assessing sample quality
QA beyond the Lab -Focal Point GS as a means of assessing sample quality
  • FP reports on the Squamous and Endocervical components detected in the sample
  • Potentially of use to the screening programme as an indicator of smear taker proficiency – but emotive!!!
  • Wales monitors TZ reporting – however generally a wide spread – between 80% - 96% reported by NW labs
  • The study has shown that the FP reported endocervical component detection rates in NW labs only varied between 79% and 85%
  • Possibly screener interpretation – CAS may provide a more uniform assessment
  • Further work is ongoing in this area
focal point gs performance across an integrated laboratory network
Focal Point GS performance across an integrated laboratory network
  • VPN data transfer means that scanned slide information is available anywhere within the combined laboratory service
  • Economies of scale are definite considerations when planning the introduction of this technology – possible consideration for LBC procurement
  • Therefore, in this operational scenario – probably more efficient to combine the sample processing and CAS on one processing site.
teaching and case review using the focal point gs
Teaching and Case review using the Focal Point GS
  • LGS can mark and record interesting features within a slide and direct the user to these areas on the slide on demand
  • With a system configured using VPN these slides can be viewed anywhere on the network
  • Considerable potential for teaching and MDT use when combined with a multi-head microscope/projector
slide30
Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles
slide31

> 15% QC

Negative

Abnormal

Sorting & Ranking (Illustration)

The FocalPoint™ SORTS and RANKS slides between the values of 0 and 1 : based on the likelihood of abnormality being present. 0 is negative : 1 is abnormal

x

< 25% Archive

> 75% Review

Q5

Q4

Q3

Q2

Q 1

fp quintile distribution

Note: this

category

Contains PR as

Well as NFR

samples

FP Quintile Distribution
slide34

Focal Point Scant Cellularity Category (FPSC)

Compared to a Final Report of Inadequate

  • Comparison of the inadequacy levels showed that the manual arm had a similar inadequate rate – not what was expected with Wales operating a 15000 cell threshold
  • Interestingly the majority of cases did not correspond between the manual and automated arms
slide35
Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples “NFR” – DOES IT WORK?
slide36

Samples Assigned for NFR

Compared to Final Result

what was missed by nfr
What was missed by NFR?
  • 1. Cases from all labs, NFR negative, finally reported as abnormal
  • 78 cases in a total of 2842
  • (Picked up by 10 screener, reviewer or checker)
  • 2.74% of cases missed by NFR – False Negatives
what was missed by nfr1
What was missed by NFR?
  • In over 50 000 cases manually rapid previewed by 4 Welsh Labs during Jan – September 2009 (Picked up by 10 screener, reviewer or checker)

The overall False Negative Rate was 3.5%

Statistically – Significant difference from 1.

NFR is therefore at least as effective as Manual Rapid Pre-screen as a QA tool for primary screening of cervical samples.

overall sensitivity
Overall Sensitivity
  • In total, the automated QC screening arm did not “identify” 12 HG cases (histologically confirmed) in a total of 134 reported

Sensitivity (134-12)/134 = 91%

  • Compare this to a quoted range of 75-95%* HG sensitivity for conventional Rapid Preview QA methods
  • 202 LG cases were missed or consigned to NFR in 788 reported LG samples

Sensitivity (788-202)/788 = 74.4%

* See references and CSW data

slide40
But……….!
  • On review, morphological abnormalities were present in the majority of 10 FOVs in the HG cases “missed “ by the automated arm.
  • So potentially therefore, 91% is the minimum sensitivity rate that can be expected
  • This is because the FocalPoint GS presented the abnormalities to the LGS user, but not identified by the user for a number of possible reasons
  • This may be due to a number of reasons e.g. unfamiliarity and further analysis currently being performed on this data
in summary
In Summary………
  • Automated QC screen appears to be no less effective than rapid pre-screen
  • Most (161/219) cases not identified by automated arm were borderline HPV cases – FP does not seem to recognise koilocytes! – especially with minimal nuclear abnormalities.
  • Despite this finding, sensitivity of the automated QC screen appears to be appreciably greater in comparison to available data on manual QC methods.
user experience is a large factor in the efficiency of the screen
User experience is a large factor inthe efficiency of the screen
  • A high proportion of the undetected abnormalities were shown to be present in the FOV’s on review – but not identified by the screener.
  • Staff had only received 6-8months experience of the technology - further work is currently ongoing to see if screener performance improved with time/experience
individual laboratory process review results
Individual LaboratoryProcess Review Results

Process Review:

Is an indicator of the number of failed slide scans in a given FP run

It reflects technical quality, bar code integrity etc.

Note the low value for YGC, which has been mainly attributed to tape coverslipping

screening times for 10 fov
Screening times for 10 FOV

Note the remarkable similarity of the mean screening times.

Note also that NFR category accounts for 22% of samples

- no FOVs, therefore potential productivity increase.

observations1
Observations
  • Thought that FP operates on TBS inadequacy threshold of 5-6k cells, therefore we hypothesised that the manual arm would identify a correspondingly higher number of inadequate samples, however……………..
    • Comparison of the inadequacy levels showed that the manual arm had only a slightly higher inadequate rate – not as expected with Wales operating a 15000 cell threshold
    • Interestingly the majority of cases did not correspond between the manual and automated arms – possibly indicative of the fact that the FPSC report is applied to cases that exhibit other criteria, such as:
observations2
Observations
  • The slide cannot be focussed on correctly because of bubbles, too thick, too few cells in the centre of the prep. etc.
  • It is therefore likely that the actual number of cases reported as FPSC for reasons related to cellular adequacy are lower than first thought
  • Further work will be undertaken to determine threshold values of the criteria that make up the FP report “Insufficient Reference Cells” to determine if this could becompared with the manual arm report of “Inadequate due to scant cellularity”
observations3
Observations
  • Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby reducing the process review rate and decreasing the number of rejected slides.
  • Selecting a large coverslip size on the tape cover slipper extended the FP scan time from approx. 11 slides/min on glass coverslips to 9 slides/min using tape coverslips. Reducing the tape size reduced this to 10 slides/min*
  • Staff used to primary screening on a x20 objective may have initial difficulties in operating the LGS which only displays FOVs at x10
  • *Note: Updated versions of the FP GS allow faster scan rates
observations4
Observations
  • Staff have varying ability to correctly interpret the 10 FOVs and like manual screening, are only as good as the experience they have – both as primary screeners and users of the LGS
  • A small proportion of slides simply cannot be evaluated on 10 FOVs – these should be subjected to a full scan and further checking, if necessary
  • Despite best efforts to blind manual and automated arms interpretational bias may have resulted as staff were aware that this was a study
observations5
Observations
  • The study resulted in additional workload for staff which might also have affected interpretation
  • Clearly defined protocols are mandatory for routine use
  • Transport of slides and data via external couriers posed a few challenges! – eventually settled down
slide54

Recommendations

  • A number of issues have yet to be fully evaluated, including:
    • Completing comparisons between available manual QC data and project results
    • Full evaluation of NFR – do we need to continue with manual QC for these cases?
    • Subsequent effects on workload/staffing
    • Implementation and logistical issues
    • Full Cost/Benefit analysis
recommendations1
Recommendations
  • Cost/benefit analysis
    • Need to further evaluate the potential of the NFR category
    • With demonstrated > 99% False Negative rate for HG disease in 22% of cases – potentially labour saving whilst improving quality!
    • This is in the region of 1100 hours screener time in Wales with an annual workload of 200 000+ cases
conclusions1
Conclusions

Analysis of the findings of this study are not yet complete.

At this stage initial results indicate that the technology is suitable for routine use, with potential quality and service management benefits to the cervical screening programme in Wales

recommendations for further investigation
Recommendations for Further Investigation
  • CSW AWMG have approved an extension of the project and in collaboration with Medical Solutions will be working towards full use of FP as a QA system. The network of laboratories will be extended to include a fourth laboratory from South Wales – at the Royal Gwent Hospital, Newport – project commenced July 2009
  • Define implemention plan
    • Networking issues – the technology seems particularly suited to networks. Wales is currently exploring the introduction of managed networks in cytology
  • Develop protocols
    • Specifically issues include standardisation of manual rapid QA methods across labs to rapid pre-screen
    • IT coding protocols require total standardisation – part of N Wales objectives. Data extraction will become more consistent
recommendations for further investigation1
Recommendations for Further Investigation
  • Implementation
    • QA use initially – follow on project to implement FP as a replacement for QC screening and further assessment as a primary screening tool, especially in the light of the changing screening programme
    • Move to full adoption subject to satisfactory outcome of project and final approval by AWMG
    • Outcome will inform LBC procurement process
slide60
“Any sufficiently advanced technology is indistinguishable from magic.”Arthur C. Clarke – The Lost Worlds of 2001 (1972)
ad