XIV PMT meeting – Dec 5, 2012
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

Agenda : Certification Platform: performance for training - I round - II round PowerPoint PPT Presentation


  • 42 Views
  • Uploaded on
  • Presentation posted in: General

XIV PMT meeting – Dec 5, 2012. Agenda : Certification Platform: performance for training - I round - II round Feedback from naives Beta testers proposals Congress presentations Paper “check for axes”. Platform: Training Phase. 2 Images (Scheltens=0,3). Feedback. Certification set.

Download Presentation

Agenda : Certification Platform: performance for training - I round - II round

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Agenda certification platform performance for training i round ii round

XIV PMT meeting – Dec 5, 2012

  • Agenda:

  • Certification Platform: performance for training - I round

  • - II round

    • Feedback from naives

  • Beta testers proposals

  • Congress presentations

  • Paper “check for axes”


  • Agenda certification platform performance for training i round ii round

    Platform: Training Phase

    2 Images (Scheltens=0,3)

    Feedback

    Certification set

    Feedback

    +

    -

    6 Images (Scheltens=0,1,2,3,4,4)

    Certification set

    10 Images (2 x Scheltens=0,1,2,3,4)

    Feedback

    +

    -


    Agenda certification platform performance for training i round ii round

    Platform: Training round I


    Agenda certification platform performance for training i round ii round

    Platform: Training rounds I and II

    Tracers received a verbal feedback in order to minimize misunderstanding possibly originated by the incomplete functionality of the platform at this stage, or due to the underlying statistical operations for the visual feedback.

    No verbal feedback for Round III


    Agenda certification platform performance for training i round ii round

    Platform: Training round I

    A qualitative check (QC) was carried out slice by slice, same procedure as for the benchmark images, but with the help of the platform:

    1) Platform usually reflects properly the correctness of segmentation. Limitations lie in CSF pools (a.: correct segmentation is not properly rendered with “green”), and in the minimum variability allowed to the naïve where masters were in great agreement with each other (b.)

    a.b.


    Agenda certification platform performance for training i round ii round

    Platform Training round I: Performance

    * Minor lacks of compliance with HP; § Major lacks of compliance with HP; §§§ More relevant segmentation mistakes than just compliance to HP


    Agenda certification platform performance for training i round ii round

    Platform Training round II: Performance

    Very atrophic sbj (Scheltens=4). Better distribute Sch=4 across training rounds next time?


    Agenda certification platform performance for training i round ii round

    Training Platform

    Tracer Area (mm^2) Area delimited by the tracer contour for the current slice.

    Master Min Area (mm^2) , Master Mean Area (mm^2) , Master Max Area (mm^2)

    Area delimited by the master minimum/mean/maximum contour for the current slice.

    Distance Integral: Line integral of the distance ratio (see definition above for “distance ratio”) values along the tracer contour, normalized by the contour’s number of point, for the current slice.

    Hausdorff Distance (mm): Hausdorff distance (see definition above) between the tracer contour and the master mean contour for the current slice.

    Dice Index w.r.t. Min, Dice Index w.r.t. Mean, Dice Index w.r.t. Max

    Dice similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice.

    Jaccard Index w.r.t. Min, Jaccard Index w.r.t. Mean, Jaccard Index w.r.t. Max

    Jaccard similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice.

    True Positive Area w.r.t. Min (mm^2), True Positive Area w.r.t. Mean (mm^2), True Positive Area w.r.t. Max (mm^2)

    Tracer true positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

    False Negative Area w.r.t. Min (mm^2), False Negative Area w.r.t. Mean (mm^2), False Negative Area w.r.t. Max (mm^2)

    Tracer false negative (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

    False Positive Area w.r.t. Min (mm^2), False Positive Area w.r.t. Mean (mm^2), False Positive Area w.r.t. Max (mm^2)

    Tracer false positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.


    Agenda certification platform performance for training i round ii round

    Certification Criteria (to be defined)

    Hypothesis so far (generated versus quality check) is that:

    Jaccard < 70 denotes major problems in hippocampal segmentation

    Jaccard < 80 denotes incomplete compliance with the HP criteria

    A hypothesis might be that at least Dice ≥ 90 and Jaccard ≥ 80 be required for qualifying.

    Anyway we first evaluate performance at third round

    (Wide statistical analysis may be required for adequate criteria selection…)


    Agenda certification platform performance for training i round ii round

    Feedback from tracers

    Would find it easier to follow a video, replacing in person communication

    (video, and also inter-linked pop-up mesg -maybe for the first two training images?- OK but later)

    For now, legends to figures and table of contents are being added to the HP to facilitate reading

    HP edited based on appropriate feedback from tracers, and re-sent to tracers

    HP will be resent to panelists after we received comments from all naïves

    Visualization improvements (are being implemented already)


    Agenda certification platform performance for training i round ii round

    b-testing

    Masami Nishikawa:

    Asks:

    access to protocol for validating VSRAD (common algorithm for hippo segmentation in Japan)

    Offers:

    Further validation branch for the SOPs project, on E-ADNI human phantoms

    (verbal offer by Matzuda of repeating segmentations on J-ADNI MRIs, having over 300 hippos segmented based on local protocol already)


    Agenda certification platform performance for training i round ii round

    Congress Presentations

    • CTAD (Montecarlo, October 29-31 2012)

    • Definition of the Harmonized Protocol for Hippocampal Segmentation

    • DGPPN Symposium (Berlin, November 21-22 2012)

    • (Andreas Fellgiebel, Stefan Teipel)

    • Segmentation of the hippocampus: Towards a joint EADC-ADNI harmonized protocol

    • Biomarkers for Brain Disorders (Cambridge, February 3-5 2013)

    • Definition of harmonized protocol for hippocampal segmentation

    • AD/PD (Florence, March 6-10 2013)

    • Definition of Harmonized Protocol for Hippocampal Segmentation

    • AAN (San Diego, March 16-23 2013)

    • EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation


    Agenda certification platform performance for training i round ii round

    Papers describing the project

    Survey of protocols (preliminary phase; Published, JAD 2011)

    Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094)

    Axes check short report (Brescia Team, to be shared with coauthors)

    Delphi consensus (Brescia Team, in progress)

    Master tracers’ practice and reliability (Brescia Team, in progr)

    Development of certification platform (Duchesne and coll)

    Validation data (Brescia Team – companion paper 1)

    Protocol definition (Brescia Team – companion paper 2)

    Validation vs pathology (TBD)

    DONEIN PROGRESSPLANNED


    Agenda certification platform performance for training i round ii round

    GOLDSTANDARD

    VALIDATION VS CURRENT PROTOCOLS

    ASSESSMENT OF SOURCES OF VARIANCE TRAINING SET DEVELOPMENT

    VALIDATION VS PATHOLOGY

    20 naïve tracers

    5 master tracers

    1 tracer

    Local Protocol

    ADNI scans: 2 x 5 Scheltens’s atrophy scores

    x 2 sides x 2 magnet strengths (1.5-3T)

    Training

    ADNI scans: 10 at 1.5T x 2 sides x 7 SUs x 2 tracing rounds

    Total per rater: 40 hippos

    QUALIFICATION

    QUALIFICATION

    Harmonized Protocol

    ADNI scans: 2 x 5 Scheltens’s atrophy score x 2 sides

    x 2 magnet strengths (1.5-3T)

    Total per rater: 40 hippos

    Harmonized Protocol

    ADNI scans: 2 x 5 Scheltens’s atrophy score x

    2 sides x 2 magnet strength (1.5-3T)

    Total per rater: 40 hippos

    Harmonized Protocol:

    Pathological datasets:

    Mayo Clinic and NYU

    Total: about 40 hippos

    Assessment of variance due to rater and center

    REFERENCE PROBABILISTIC MASKS

    with 95% C.I.

    Best 5 naïve tracers

    Assessment of agreement with volume on pathology or ex vivo MRI and correlation with neuronal density

    Harmonized Protocol

    ADNI scans: 2 sides x 5 Scheltens’s atrophy scores x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl)

    x 2 magnet strengths (1.5-3T)

    Total per rater: 240 hippos

    Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater

    TRAINING SET


    Agenda certification platform performance for training i round ii round

    GANTT


  • Login