Program Update October 11, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench

1. Program Update October 11, 2012 Andrew J. Buckler, MS Principal Investigator, QI-Bench With Funding Support provided by National Institute of Standards and Technology

2. Agenda • Summary and close-out of FY12 development iteration • Coveringwhat’sbeenaccomplishedfrom multiple pointsofview • FY13 development iteration • Deployment progress and support • Continued Progress on ISA files • Architecture • Contour-based analysis 2

3. FY 2012(n=110) Autumn 2012(n=54) • Establish overall structure • Initial Specify, Formulate, and Iterate • Substantial work in Execute and Analyze • First work on test-beds • V&V issues in Execute • Substantial work in Analyze library • Deployment support • Advance test-beds Winter 2013(n=15) In Queue(n=6) • Introduce radiologist workstation component, including scripted reader studies • First real implementation of Biomarker KB triple store • W3C-compliance in Specify • Formulate using SPARQL • Full realization of QI-Bench cohesive architecture • Full realization of worked example test-bed 3 3 3

4. V&V • QI-Bench Project Management Plan (PMP) • Traceability Report • QI-Bench Verification and Validation Plan • QI-Bench Iteration 1 Validation Report • QI-Bench Iteration 1 Verification Protocol • QI-Bench Iteration 1 Verification Report • Application Test Protocols, Reports, and Records: • Specify: from AIM • Formulate: from caB2B • Execute • From MIDAS • RDSM Integration Test Report • Analyze • From AVT • Library Integration Test Report • Iterate: from Taverna Lab Protocol Design Documents User Needs and Requirements Analysis Architecture Application-specific Design Specify "Specify" Scope Description (ASD) "Specify" Architecture Specification (AAS) "Quantitative Imaging Biomarker Ontology (QIBO)" Software Design Document (SDD) "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD) AIM Template Builder Design Documentation: Formulate "Formulate" Scope Description (ASD) "Formulate" Architecture Specification (AAS) "NBIA Connector" Software Design Document (SDD) Execute "Execute" Scope Description (ASD) "Execute" Architecture Specification (AAS) Reference Data Set Manager (RDSM) Software Design Document (SDD) Batch Analysis Service Software Design Document (SDD) Analyze "Analyze" Scope Description (ASD) "Analyze" Architecture Specification (AAS) Package "Package" Scope Description (ASD) "Package" Architecture Specification (AAS) • Develop and run queries based on data requirements • Use of Formulate • Load Reference Data into the Reference Data Set Manager • Example Pilot3A Data Processing Steps • Server-Side Processing using the Batch Analysis Service • Package Algorithm or Method using Batch Analysis Service API • Prepare Data Set • Create Ground Truth or other Reference Annotation and Markup • Importing location points and other data for use • Writing Scripts • Initiate a Batch Analysis Run • Perform statistical analysis • Analyze Use Instructions 4 4 4

5. Demonstrator 15, 40, 3A Pilot and Pivotal, and Change Analysis … Challenge Definition: estimate absolute volumes in phantom data Explicitly indicate descriptive statistics: bias, variance. Null hypothesis:analysis software model does not have a significant effect on the bias and variance. Phantom data, FDA, NIST, QI-Bench FDA, M. A Gavrielides et al., “A resource for the Assessment of lung nodule size estimation methods: database of thoracic CT scans of an anthropomorphic phantom”, Optics Express, vol. 18, n.14, pp. 15244- 15255, 2010. • Investigation 1: • Pilot and pivotal study are finished PIVOTAL STUDY PILOT STUDY 12 participants who measured 97 nodules 10 participants who measured 408 nodules Fig. 1: Radial plot showing comparative performance on the selected descriptive statistics as well as mean of absolute percent errors. 5 5 5

6. 6 6 6

7. 7 7 7

8. FY13 Development Iteration • Deployment progress and support • Continued Progress on ISA files • Architecture • Contour analysis (purpose, methods, and file formats) 8 8 8 8

9. Continued Progress on ISA Files • Assay (a_) and Study (s_) levels: • _dcm.csv: SUBJID, TPINDEX, SITE, ACQREP, SERIESTYPE, <dicom fields> • _loc.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, locX, locY, locZ, bbX0, bbX1, bbY0, bbY1, bbZ0, bbZ1 • _rdg.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, SEGRDR, SEGTOOL, SEGREP, SERIESTYPE, READING, <project specific fields> • _cov.csv: SUBJID, AGE, GENDER, HEIGHT, WEIGHT, RACE, <project specific fields> • _chg.csv: SUBJID, TARGET, TPINDEX1ST, ACQREP1ST, VALUE1ST, TPINDEX2ND, ACQREP2ND, VALUE2ND, ARITHDIFF, PCTDIFF, PPNDIFF, <project specific fields> • _dx.csv: SUBJID, TARGET, deltaX, X, SOURCE • _mo.csv: TYPE, INSTANCE, VALUE, MODULE • Investigation (i_) level: • Works in progress, but concept is serialized triple store roll-up including aggregation analyses such as aggregate uncertainty 9 9 9 9

10. Web GUI • High level features: • GWT (or Tapestry) UI ; both desktop and web client versions • RESTful service layer; need to work out details between Hibernate and Jena • Implemented according to open source best practices; • In such a way as to enable the enhancement roadmap; and • Integrated with projects driving advanced semantics and support for regulatory e-submissions Applications (a generic QI-Bench template, as well as specific configurations) QI-Bench Stack XIP LIB Taverna R LIB Cached objects: AIM/DICOM, etc Data Access Layer Modified XIP Host Jena Hibernate RDSM for Images and ISA files RDF triple store for Patient info, annotations, Collections Experiments data services (e.g., MIDAS, NBIA, etc.) Ontologies and vocabularies 10

11. Cohort applications: • Specify • Formulate • Execute’s RDSM and BAS • Analyze/Iterate (may combine) Applications (a generic QI-Bench template, as well as specific configurations) XIP LIB Taverna R LIB Cached objects: AIM/DICOM, etc Data Access Layer 11

12. Individual Subject (Patient) Workstation • Plug-in to <fill in your favorite workstation> • We provide: • wrapper for ClearCanvas as example and template • Data access layer: • Unified worklist transactions • Support for Q/R to RDSM • Support for access to Biomarker KB • Taverna-desktop level of capability for workflows Applications (a generic QI-Bench template, as well as specific configurations) XIP LIB Taverna R LIB Cached objects: AIM/DICOM, etc Data Access Layer Java core, for consistency across QI-Bench? C++ layer, for leverage of components in C++ and support of ClearCanvas? 12

13. GUI • Local Host (“workstation”) configuration (thick client) local Modified XIP Host Jena Hibernate remote RDSM for Images and ISA files RDF triple store for Patient info, annotations, Collections Experiments data services (e.g., MIDAS, NBIA, etc.) Ontologies and vocabularies 13

14. Web GUI • Web-based (thin client) local Modified XIP Host Jena Hibernate remote RDSM for Images and ISA files RDF triple store for Patient info, annotations, Collections Experiments data services (e.g., MIDAS, NBIA, etc.) Ontologies and vocabularies 14

15. Early ideas on Technical Approach • What • Re-architect UI for Analyze application • Interface XIP Host Services through REST API • Retrieve AIM data and pass to Analyze • Present analysis data in new web UI • Interface with all AIM and DICOM data through XIP Host Services • How • Incorporate Hibernate object-relational mapping (ORM) for DB2, Midas • Build XIP Host Services instance • Retrieve all data through XIP Host Services and WADO • Select Web application Framework GWT, Tapestry, Spring, Wicket, HybridJava, etc., and build UI for • XIP Host Services for Data Retrieval • presenting results from MVT application • interacting with MVT analysis data based on User Stories and Use Cases. • Using Hibernate, persist results using Jena KB 15

16. Contour-based Analysis • Purpose • Methods: • STAPLE • Meyer’s P-maps • MICCAI indices • DICE • File formats: • DICOM segmentation objects • AIM 4.0 • STL • MHT 16 16 16 16

17. 17

18. Value proposition of QI-Bench • Efficiently collect and exploit evidence establishing standards for optimized quantitative imaging: • Users want confidence in the read-outs • Pharma wants to use them as endpoints • Device/SW companies want to market products that produce them without huge costs • Public wants to trust the decisions that they contribute to • By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data • Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders 18

19. Summary:QI-Bench Contributions • We make it practical to increase the magnitude of data for increased statistical significance. • We provide practical means to grapple with massive data sets. • We address the problem of efficient use of resources to assess limits of generalizability. • We make formal specification accessible to diverse groups of experts that are not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or requirements within specific contexts for use. • We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 19

20. QI-BenchStructure / Acknowledgements • Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna Danagoulian) • Co-Investigators • Kitware (Rick Avila, Patrick Reynolds, JulienJomier, Mike Grauer) • Stanford (David Paik) • Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu) • Collaborators / Colleagues / Idea Contributors • Georgetown (Baris Suzek) • FDA (Nick Petrick, Marios Gavrielides) • UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha) • Northwestern (Pat Mongkolwat) • UCLA (Grace Kim) • VUmc (Otto Hoekstra) • Industry • Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner) • Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, … • Coordinating Programs • RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao) • Under consideration: CTMM TraIT (Andre Dekker, JeroenBelien) 20

21. Statistical Validation Service for Imaging Quantitatively characterizing and optimizing performance of imaging accelerates discovery and widens the availability of new treatments to patients with unmet medical needs. • We bring MVT forward as it could be beyond what it currently is; • Available as a web application with thin and thick-client options with persistent database. • Implemented to be generalizable to needs of RadOnc, QIN, QIBA, FNIH, C-Path, and other members of the community.  Applications include: • Augmenting the current genome based biomarkers with imaging based biomarkers in Transcend for Breast cancer and/or TCGA for brain cancer • Community Cancer Centers • Project may be synergistically pursued with the FDA imaging submission project. 21