slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure PowerPoint Presentation
Download Presentation
Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

Loading in 2 Seconds...

play fullscreen
1 / 30

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure - PowerPoint PPT Presentation

  • Uploaded on

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure. Anthony Kolasny Johns Hopkins University. Center for Imaging Science. Institute for Computational Medicine. Computational Anatomy.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure' - ankti

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Computational Anatomy: Utilizing BIRN and TeraGrid Infrastructure

Anthony Kolasny

Johns Hopkins University


Computational Anatomy

  • Computational Anatomy's goal is to define methods for the quantization of shape within biological structures.
  • Origins of Computational Anatomy (CA) may be found in the central thesis of Sir D'Arcy Wentworth Thompson's 1917 book entitled On Growth and Form.

D'Arcy believed that biologists of his day over emphasized the role of evolution, and under emphasized the roles of physical laws and mechanics, as determinants of the form and structure of living organisms.


Deformable Templates

Image from D'arcy Thompson "On Growth and Form"

"Computational anatomy: an emerging discipline," Ulf Grenander

and Michael I. Miller (Quart. Appl. Math. 56[4]: 617-94, December 1998)‏


Large Deformation Diffeomorphic Metric Mapping

  • The Large Deformation Diffeomorphic Metric Mapping (LDDMM) is an application which computes a metric distance on the space of anatomical images in Computational Anatomy thereby allowing for the direct comparison and quantization of morphometric changes in shapes.

Metric Distance

a metric or distance function is a function which defines a distance

between elements of a set.




Multidimensional Scaling (MDS)‏

Metric multidimensional scalinng (MDS) 

A superset of classical MDS that generalizes the optimization procedure to a variety of loss functions and input matrices of known distances with weights and so on.

Linear discriminant analysis (LDA)‏

Used in statistics to find the linear combination of features which best separate two or more classes of object or event.


Biomedical Informatics Research Network (BIRN)‏

BIRN is a National Center for Research Resources (NCRR) initiative aimed at creating a testbed to address biomedical researchers







Shape Analysis - A Morphometry BIRN Project

Data Donor





And upload



Shape Analysis

of Segmented Structures





Goal: comparison and quantification of structures’ shape and volumetric differences across patient populations


Identifying Shape Analysis Requirements

  • 2002 – Collecting Requirements
  • Identify the testbed process - WashU/MGH/JHU/BWH
  • Identifying the population – Normal/Alzheimer's/Sematic Dementia
  • LDDMM only runs on an IBM SP (96 cpus, we use 8 cpus per job)
  • Each job takes about 8 hrs.
  • 101 datasets
  • We need to compare right/left unscaled/scaled (40,804 jobs)‏
  • 40TB (cost for storage $300K)‏
  • 326,432 cpu/hrs (37.2 cpu/years or 3.1 years on the IBM SP)‏


Leverage BIRN SRB Storage

BIRN + TeraGrid would write to a common area

Port LDDMM to utilize TeraGrid

Work with Intel to optimize code. (30% speedup)‏

Provide a portal interface to LDDMM

Allow other people access to the program

Utilize VTK for visualization

Utilize Wikis for documentation

Leveraging the BIRN Infrastructure

first iteration

First & Second Iteration of Shape Analysis Project

First Iteration

Second Iteration

  • Processed 18 data sets
  • 1xN comparisons
  • Utilized IBM SP
  • Local Storage
  • Output VTK
  • Non-Conclusive
  • Processed 45 data sets
  • NxN comparisons
  • Utilized TeraGRid
  • Utilized SRB
  • Visualized VTK
  • Emerging Classifier

Classification (45 Train only)‏


1: Control

2: Alzheimer’s

3: SD



Preparing for the 101 Data Set

  • Already did 45x45
  • 32,704 jobs to do.
  • 200,000 cpu/hrs – Utilize SDSC, NCSA, BIRN, JHU
  • 32 TB Storage – Utilize GPFS-WAN
  • Glue clusters and storage using sshfs
  • Implement a simple queue to submit to the various queueing systems.
  • Visualize with Paraview, Mayvi and 3D Slicer

Classification (45 Train + 56 Test)‏


1: Control

2: Alzheimer’s

3: SD


•: Control

▲: Alzheimer’s

* : mean(1,•)‏

* : mean(2,3,▲)‏



Statistical Inference

2 class

56 Test points

0.0042 ± 0.001

  • Estimated p-values of a permutation test
  • Inference: The original LDDMM operation captures shape information in the MR images that is correlated with clinical diagnoses.

Publications: M. Miller, C. Priebe, B. Fischl, A. Kolasny, Y. Park, E. Busa, J. Jovivich, P. Yu, B. Dickerson, R. Buckner, Morphometry BIRN , Collaborative Computational Anatomy:The Perfect Storm for MRI Morphometric Study of the Human Brain via Diffeomorphic Metric Mapping, Multidimensional Scaling and Linear Disriminant Analysis, Proceedings of the National Academy of Sciences - Submitted for review


Secure Shell FileSystem (SSHFS)‏

a file system for Linux (and other operating systems with a FUSE implementation, such as Mac OS X) capable of operating on files on a remote computer using just a secure shell login on the remote computer. On the local computer where the SSHFS is mounted, the implementation makes use of the FUSE (Filesystem in Userspace) kernel module. The practical effect of this is that the end user can seamlessly interact with remote files being securely served over SSH just as if they were local files on his/her computer.

The current implementation of SSHFS using FUSE is a rewrite of an earlier version. The rewrite was done by Miklos Szeredi, who also wrote FUSE.

sshfs remoteuser@remotehost:/path/to/remote_dir local_mountpoint


Autofs and SSHFS

Autofs allows automatic mounting of remote sshfs filesytems. CIS users may access remote data as local directories. Common local directory structure allows for more effective scripting and analysis of data.


Gluing Clusters and Storage with SSHFS

Utilizing samba and smbwebclient, we were able create a web interface to clustered data.


Common Data Namespace Improved Throughput

/gpfs-wan is shared at SDSC and NCSA in a common location. Using sshfs, /gpfs-wan was mounted on BIRN SDSC Cluster and the BIRN JHU cluster adding more processing power.

  • run – Contains the list of jobs for the queues at SDSC, NCSA, BIRN, JHU
  • submit_q_script_ncsa – monitors qstatus and adds more jobs on an hourly basis
  • ncsa1 – Contains current jobs running in the NCSA queue
  • done – when job is finished, send it to the done directory

Monitoring the Queues

Monitoring the SDSC, NCSA, JHU and BIRN clusters is as simple as reading

email. Monitoring what's left to run is as easy as 'cd run; ls | wc'.



Wikis are extremely helpful in keeping track of projects and monitor progress



Utilizing Paraview we are able to

visualize structures and velocity data.



  • Provides access to lddmm
  • Utilized BIRN Infrastructure
$300K - $40K = Saved $260K in storage costs.

Invested $70k in development cluster. Current cumulative TeraGrid time 640K cpu/hrs.

TeraGrid help desk an extremely valuable asset.




2002.09 - BIRN All Hands

2002.10 - Supercomputing Baltimore emerging TeraGrid

2003.02 - Morph BIRN - define SASHA project

2003.05 - BIRN Rack installed

2003.08 - lddmm writes vtk output

2003.09 - Installed itanium2 cluster

2003.09 - lddmm processing on IBM SP (18 subjects)‏

2003.10 - BIRN All Hands – Shape Analysis Pipeline (not conclusive from 18 subjects)‏

2004.02 - Morph BIRN All Hands

2004.02 - 40K cpu/hrs award 45x45 processing used SRB

2004.06 - Human Brain Mapping - Emerging Classifier

2004.08 - Intel HPC workshop 30% performance improvement

2005.09 - 300K cpu/hrs award

2006.01 - Started processing 101x101 processing using GPFS-WAN

2006.06 - Human Brain Mapping Conference – Present lddmm, MDS, LDA

2006.09 - Submitted PNAS paper

2006.10 - SDSC Calendar highlights BIRN Shape Analysis Project

2006.09 - 300K cpu/hrs other lddmm projects Mouse BIRN, OASIS, VETSA, ADNI




CIS Members

Michael I. Miller

Carey Priebe

Can Ceritoglu

Timothy Brown

Youngser Park

CIS Alumni

Faisal Beg

BIRN Collaborators

Mark Ellisman

Steve Pieper

Bruce Fischl

Randy Buckner

TeraGrid Support

Amitava Majumdar

Nancy Wilkins-Diehr

Thank You