1 / 27

Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications

Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications. Vladimir Korkhov 1 , Dagmar Krefting 2 , Tamas Kukla 3 , Gabor Terstyanszky 3 , Matthan Caan 1 and Silvia Olabarriaga 1 1 Academic Medical Center of the University of Amsterdam, NL

jcore
Download Presentation

Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using SHIWA Workflow Interoperability Tools for Neuroimaging Data Analysis Applications Vladimir Korkhov1, Dagmar Krefting2, Tamas Kukla3, Gabor Terstyanszky3, Matthan Caan1 and Silvia Olabarriaga1 1Academic Medical Center of the University of Amsterdam, NL 2Charité - Universtätsmedizin Berlin, DE 3Centre for Parallel Computing, University of Westminster, UK EGI Community Forum, Munich, March 28, 2012

  2. Outline • Motivation • SHIWA platform • Coarse-grained interoperability • SHIWA Simulation Platform: repository & portal • Neuroimaging use-case • Workflow implementations • Interoperability experiments • Conclusion EGI Community Forum, Munich, March 28, 2012

  3. Motivation • User perspective: applications (workflows) developed at different organizations • Share own, re-use workflows of others • Create and execute meta-workflows: workflow interoperability • Get more resources for own workflows by running on a different DCI (or several DCI at the same time) • Different DCIs and workflow systems! DCI – distributed computing infrastructure EGI Community Forum, Munich, March 28, 2012

  4. Motivation: generic use-case SE CE DCI B SE CE DCI A DCI – distributed computing infrastructure; CE/SE – computing/storage element EGI Community Forum, Munich, March 28, 2012

  5. Motivation: generic use-case find and re-use combine WFs with own data SE CE on own CE? DCI B SE CE DCI A DCI – distributed computing infrastructure; CE/SE – computing/storage element EGI Community Forum, Munich, March 28, 2012

  6. What is SHIWA? SHIWA: SHaringInteroperableWorkflows for Large-ScaleScientific Simulations on Available DCIs http://www.shiwa-workflow.eu • Workflow interoperability • Sharing and re-using workflows • Creating and executing meta-workflows • Utilizing heterogeneous DCIs (together) EGI Community Forum, Munich, March 28, 2012

  7. Target: SHIWA Science Gateway CGI – coarse-grained interoperability; FGI – fine-grained interoperability;IWIR - Interoperable Workflow Intermediate Representation EGI Community Forum, Munich, March 28, 2012

  8. Coarse-grained interoperability • CGI = Nesting of different workflow systems to achieve interoperability of execution frameworks EGI Community Forum, Munich, March 28, 2012

  9. SHIWA Simulation Platform/CGI EGI Community Forum, Munich, March 28, 2012

  10. SSP/CGI: Infrastructures and VOs EGI Community Forum, Munich, March 28, 2012

  11. Use case: Neuroimaging • Magnetic Resonance Imaging (MRI) • Diffusion Tensor MRI (DTI) • Magnetic Resonance imaging modality enabling the identification of the orientation of human tissue. • Indirect measure of water diffusion in brain tissue • Used in comparative studies of brain diseases that are thought to cause local damage to brain tissue • Data analysis workflows: • DTI-preprocessing (AMC) • FSL BedpostX (Charité and AMC) EGI Community Forum, Munich, March 28, 2012

  12. Workflows: DTI pre-processing • DTI-preprocessing (AMC) • Remove artifacts, noise, movement • In-house software based on Matlab • Inputs: DICOM, NIfTY, PARREC data • Output: multiple formats including bedpostX • Workflow • MOTEUR WF engine, GWENDIA • VleMed VO, ported to SHIWA VO EGI Community Forum, Munich, March 28, 2012

  13. Workflows: BedpostX • FSL BedpostX (Charité) • Reconstruct brain fibers, detect crossings • Based on FMRIB Software Library (FSL) • Inputs: image and metadata in specific formats • Outputs: directory with results and statistics • Workflows • GWES and MOTEUR (AMC) • MediGrid VO, ported to SHIWA VO EGI Community Forum, Munich, March 28, 2012

  14. Interoperability scenarios • Publish executable workflows in the repo • Find and test workflows • Run workflows with own data • Create and execute meta-workflows • Perform parallel processing on multiple DCIs EGI Community Forum, Munich, March 28, 2012

  15. Execution using SHIWA services • Used SHIWA repository to: • Describe workflows • Share workflows • Used SHIWA portal to: • Access and enact registered workflows • Compose and enact meta-workflows • Monitor workflows and meta-workflows execution • Retrieve results of the execution EGI Community Forum, Munich, March 28, 2012

  16. Scenario 1: Publish WFs • SHIWA Repository • Provide WF description, describe inputs and outputs of WFs, upload files for implementations, and sample inputs/outputs (“configurations”) EGI Community Forum, Munich, March 28, 2012

  17. Scenario 2: Find and test WFs • SHIWA Repository: Analyze description, inputs and outputs of published WFs • SHIWA Portal: Instantiate WF from repo, execute with given sample data(inside P-GRADE workflow used as the Master WF system) EGI Community Forum, Munich, March 28, 2012

  18. Scenario 3: Run WFs with own data • SHIWA Portal: Instantiate WF from SHIWA repo, execute with own data: • on the same VO/DCI • on different VO/DCI (inter-DCI transfers needed) copy data to execution VO generate WF inputs execute WF copy data from execution VO parse WF output EGI Community Forum, Munich, March 28, 2012

  19. Scenario 4: Create and run meta-workflows • SHIWA Portal: several WFs running on different DCIs • Feed output of one to input of another • Different DCIs: data transfers of intermediate data needed execute WF1 process WF1 out copy data generate WF2 in execute WF2 EGI Community Forum, Munich, March 28, 2012

  20. FSL-BedpostX MOTEUR/EGI/SHIWA VO FSL-BedpostX GWES/D-Grid Scenario 5: Parallel processing on different DCIs • Meta-workflow with split-process-merge pattern • Processing in parallel with different implementations of WF EGI Community Forum, Munich, March 28, 2012

  21. Scenario 5: Parallel processing on different DCIs Inter-VO/DCI data transfers! prepare data copy data gen WF in gen WF in execute WF execute WF parse WF out copy data collect res parse WF out EGI Community Forum, Munich, March 28, 2012

  22. Execution in SHIWA Portal EGI Community Forum, Munich, March 28, 2012

  23. Discussion / future work • Use of multiple DCIs: • handling multiple user credentials • data transfers • Data exchange between sub-workflows • Control of data formats • Error handling • Provenance EGI Community Forum, Munich, March 28, 2012

  24. SHIWA for users Re-use others’ workflows from other WF systems/DCIs Get more resources for your WFs, transparently utilize multiple DCIs Compose heterogeneous meta-workflows for complex multi-step analysis Share your workflows with others so that they can use them right away even in different environment EGI Community Forum, Munich, March 28, 2012

  25. Conclusion • Sharing and re-use of interoperable workflows needed to promote scientific collaboration • SHIWA provides a promising platform: • repository and portal • desktop and compilers coming • Performed workflow interoperability experiments evaluate SHIWA solutions • SHIWA is in progress • still much to be done but the current solution already facilitates workflow sharing EGI Community Forum, Munich, March 28, 2012

  26. Acknowledgements • Johan Montagnat, CNRS • Tristan Glatard, CNRS • Tram Truong Huu, CNRS • Sarra Ben Fredj, CNRS • Mario Wohlfahrt, Charite • ShayanShahand, AMC • Mark Santcroos, AMC This work makes use of results produced by the SHIWA, an Integrated Infrastructure Initiative (I3) project co-funded by the European Commission (under contract number 261585) through the Seventh Framework Programme. EGI Community Forum, Munich, March 28, 2012

  27. SHIWA Tutorial Friday, March 30 11.00-12.30 & 14.00-15.30 LRZ2 EGI Community Forum, Munich, March 28, 2012

More Related