Computational Anatomy: Utilizing BIRN and TeraGrid
This presentation is the property of its rightful owner.
Sponsored Links
1 / 30

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure PowerPoint PPT Presentation


  • 62 Views
  • Uploaded on
  • Presentation posted in: General

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure. Anthony Kolasny Johns Hopkins University. Center for Imaging Science. Institute for Computational Medicine. Computational Anatomy.

Download Presentation

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Computational anatomy utilizing birn and teragrid infrastructure

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

Anthony Kolasny

Johns Hopkins University


Center for imaging science

Center for Imaging Science


Computational anatomy utilizing birn and teragrid infrastructure

Institute for Computational Medicine


Computational anatomy utilizing birn and teragrid infrastructure

Computational Anatomy

  • Computational Anatomy's goal is to define methods for the quantization of shape within biological structures.

  • Origins of Computational Anatomy (CA) may be found in the central thesis of Sir D'Arcy Wentworth Thompson's 1917 book entitled On Growth and Form.

D'Arcy believed that biologists of his day over emphasized the role of evolution, and under emphasized the roles of physical laws and mechanics, as determinants of the form and structure of living organisms.


Computational anatomy utilizing birn and teragrid infrastructure

Deformable Templates

Image from D'arcy Thompson "On Growth and Form"

"Computational anatomy: an emerging discipline," Ulf Grenander

and Michael I. Miller (Quart. Appl. Math. 56[4]: 617-94, December 1998)‏


Computational anatomy utilizing birn and teragrid infrastructure

Large Deformation Diffeomorphic Metric Mapping

  • The Large Deformation Diffeomorphic Metric Mapping (LDDMM) is an application which computes a metric distance on the space of anatomical images in Computational Anatomy thereby allowing for the direct comparison and quantization of morphometric changes in shapes.


Computational anatomy utilizing birn and teragrid infrastructure

Metric Distance

a metric or distance function is a function which defines a distance

between elements of a set.

finalMetricDistance.txt

6.4898851


Computational anatomy utilizing birn and teragrid infrastructure

Multidimensional Scaling (MDS)‏

Metric multidimensional scalinng (MDS) 

A superset of classical MDS that generalizes the optimization procedure to a variety of loss functions and input matrices of known distances with weights and so on.

Linear discriminant analysis (LDA)‏

Used in statistics to find the linear combination of features which best separate two or more classes of object or event.


Computational anatomy utilizing birn and teragrid infrastructure

Biomedical Informatics Research Network (BIRN)‏

BIRN is a National Center for Research Resources (NCRR) initiative aimed at creating a testbed to address biomedical researchers


Computational anatomy utilizing birn and teragrid infrastructure

4

3

5

TeraGrid

Supercomputing

Shape Analysis - A Morphometry BIRN Project

Data Donor

Sites

1

Storage

De-identification

And upload

2

JHU CIS-KKI

Shape Analysis

of Segmented Structures

MGH

Segmentation

BWH

Visualization

Goal: comparison and quantification of structures’ shape and volumetric differences across patient populations


Computational anatomy utilizing birn and teragrid infrastructure

Identifying Shape Analysis Requirements

  • 2002 – Collecting Requirements

  • Identify the testbed process - WashU/MGH/JHU/BWH

  • Identifying the population – Normal/Alzheimer's/Sematic Dementia

  • LDDMM only runs on an IBM SP (96 cpus, we use 8 cpus per job)

  • Each job takes about 8 hrs.

  • 101 datasets

  • We need to compare right/left unscaled/scaled (40,804 jobs)‏

  • 40TB (cost for storage $300K)‏

  • 326,432 cpu/hrs (37.2 cpu/years or 3.1 years on the IBM SP)‏

HELP!


Computational anatomy utilizing birn and teragrid infrastructure

Leverage BIRN SRB Storage

BIRN + TeraGrid would write to a common area

Port LDDMM to utilize TeraGrid

Work with Intel to optimize code. (30% speedup)‏

Provide a portal interface to LDDMM

Allow other people access to the program

Utilize VTK for visualization

Utilize Wikis for documentation

Leveraging the BIRN Infrastructure


Computational anatomy utilizing birn and teragrid infrastructure

TeraGrid


First iteration

First & Second Iteration of Shape Analysis Project

First Iteration

Second Iteration

  • Processed 18 data sets

  • 1xN comparisons

  • Utilized IBM SP

  • Local Storage

  • Output VTK

  • Non-Conclusive

  • Processed 45 data sets

  • NxN comparisons

  • Utilized TeraGRid

  • Utilized SRB

  • Visualized VTK

  • Emerging Classifier


Computational anatomy utilizing birn and teragrid infrastructure

Classification (45 Train only)‏

Training

1: Control

2: Alzheimer’s

3: SD

{


Computational anatomy utilizing birn and teragrid infrastructure

Preparing for the 101 Data Set

  • Already did 45x45

  • 32,704 jobs to do.

  • 200,000 cpu/hrs – Utilize SDSC, NCSA, BIRN, JHU

  • 32 TB Storage – Utilize GPFS-WAN

  • Glue clusters and storage using sshfs

  • Implement a simple queue to submit to the various queueing systems.

  • Visualize with Paraview, Mayvi and 3D Slicer


Computational anatomy utilizing birn and teragrid infrastructure

Classification (45 Train + 56 Test)‏

Training

1: Control

2: Alzheimer’s

3: SD

Testing

•: Control

▲: Alzheimer’s

* : mean(1,•)‏

* : mean(2,3,▲)‏

{


Computational anatomy utilizing birn and teragrid infrastructure

Statistical Inference

2 class

56 Test points

0.0042 ± 0.001

  • Estimated p-values of a permutation test

  • Inference: The original LDDMM operation captures shape information in the MR images that is correlated with clinical diagnoses.

Publications: M. Miller, C. Priebe, B. Fischl, A. Kolasny, Y. Park, E. Busa, J. Jovivich, P. Yu, B. Dickerson, R. Buckner, Morphometry BIRN , Collaborative Computational Anatomy:The Perfect Storm for MRI Morphometric Study of the Human Brain via Diffeomorphic Metric Mapping, Multidimensional Scaling and Linear Disriminant Analysis, Proceedings of the National Academy of Sciences - Submitted for review


Computational anatomy utilizing birn and teragrid infrastructure

Secure Shell FileSystem (SSHFS)‏

a file system for Linux (and other operating systems with a FUSE implementation, such as Mac OS X) capable of operating on files on a remote computer using just a secure shell login on the remote computer. On the local computer where the SSHFS is mounted, the implementation makes use of the FUSE (Filesystem in Userspace) kernel module. The practical effect of this is that the end user can seamlessly interact with remote files being securely served over SSH just as if they were local files on his/her computer.

The current implementation of SSHFS using FUSE is a rewrite of an earlier version. The rewrite was done by Miklos Szeredi, who also wrote FUSE.

sshfs [email protected]:/path/to/remote_dir local_mountpoint


Computational anatomy utilizing birn and teragrid infrastructure

Autofs and SSHFS

Autofs allows automatic mounting of remote sshfs filesytems. CIS users may access remote data as local directories. Common local directory structure allows for more effective scripting and analysis of data.


Computational anatomy utilizing birn and teragrid infrastructure

Gluing Clusters and Storage with SSHFS

Utilizing samba and smbwebclient, we were able create a web interface to clustered data.


Computational anatomy utilizing birn and teragrid infrastructure

Common Data Namespace Improved Throughput

/gpfs-wan is shared at SDSC and NCSA in a common location. Using sshfs, /gpfs-wan was mounted on BIRN SDSC Cluster and the BIRN JHU cluster adding more processing power.

  • run – Contains the list of jobs for the queues at SDSC, NCSA, BIRN, JHU

  • submit_q_script_ncsa – monitors qstatus and adds more jobs on an hourly basis

  • ncsa1 – Contains current jobs running in the NCSA queue

  • done – when job is finished, send it to the done directory


Computational anatomy utilizing birn and teragrid infrastructure

Monitoring the Queues

Monitoring the SDSC, NCSA, JHU and BIRN clusters is as simple as reading

email. Monitoring what's left to run is as easy as 'cd run; ls | wc'.


Computational anatomy utilizing birn and teragrid infrastructure

Wikis

Wikis are extremely helpful in keeping track of projects and monitor progress


Computational anatomy utilizing birn and teragrid infrastructure

Visulalization

Utilizing Paraview we are able to

visualize structures and velocity data.


Computational anatomy utilizing birn and teragrid infrastructure

BIRN LDDMM Portal

  • Provides access to lddmm

  • Utilized BIRN Infrastructure


Computational anatomy utilizing birn and teragrid infrastructure

$300K - $40K = Saved $260K in storage costs.

Invested $70k in development cluster. Current cumulative TeraGrid time 640K cpu/hrs.

TeraGrid help desk an extremely valuable asset.

Savings


Computational anatomy utilizing birn and teragrid infrastructure

Timeline

2002.09 - BIRN All Hands

2002.10 - Supercomputing Baltimore emerging TeraGrid

2003.02 - Morph BIRN - define SASHA project

2003.05 - BIRN Rack installed

2003.08 - lddmm writes vtk output

2003.09 - Installed itanium2 cluster

2003.09 - lddmm processing on IBM SP (18 subjects)‏

2003.10 - BIRN All Hands – Shape Analysis Pipeline (not conclusive from 18 subjects)‏

2004.02 - Morph BIRN All Hands

2004.02 - 40K cpu/hrs award 45x45 processing used SRB

2004.06 - Human Brain Mapping - Emerging Classifier

2004.08 - Intel HPC workshop 30% performance improvement

2005.09 - 300K cpu/hrs award

2006.01 - Started processing 101x101 processing using GPFS-WAN

2006.06 - Human Brain Mapping Conference – Present lddmm, MDS, LDA

2006.09 - Submitted PNAS paper

2006.10 - SDSC Calendar highlights BIRN Shape Analysis Project

2006.09 - 300K cpu/hrs other lddmm projects Mouse BIRN, OASIS, VETSA, ADNI


Computational anatomy utilizing birn and teragrid infrastructure

http://en.wikipedia.org/wiki/D'Arcy_Thompson

http://en.wikipedia.org/wiki/Pattern_theory

http://www.cis.jhu.edu/software

http://www.cis.jhu.edu/portal/birn/

http://www.cis.jhu.edu/software/lddmm/clinical.html

http://en.wikipedia.org/wiki/Multidimensional_scaling

http://www.analytictech.com/borgatti/papers/Visualizing_proximities.pdf

http://www.nbirn.net/press/archive/ahm_2006/ppts/sshfs_ahm2006.ppt

http://en.wikipedia.org/wiki/SSHFS

http://smbwebclient.sourceforge.net/

http://paraview.org/ http://www.slicer.org/

http://www.sdsc.edu/News%20Items/PR100606.html

References


Contributors

Contributors

CIS Members

Michael I. Miller

Carey Priebe

Can Ceritoglu

Timothy Brown

Youngser Park

CIS Alumni

Faisal Beg

BIRN Collaborators

Mark Ellisman

Steve Pieper

Bruce Fischl

Randy Buckner

TeraGrid Support

Amitava Majumdar

Nancy Wilkins-Diehr

Thank You


  • Login