1 / 29

Toward interactive visualization in a distributed workflow

Toward interactive visualization in a distributed workflow. Steven G. Parker Oscar Barney Ayla Khan Thiago Ize. Component-Based Architectures. Experience with numerous component-based architectures CCA (Parallel, Method Invocation, multi-language) SCIRun (Shared memory, Dataflow, C++)

cianna
Download Presentation

Toward interactive visualization in a distributed workflow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize

  2. Component-Based Architectures • Experience with numerous component-based architectures • CCA (Parallel, Method Invocation, multi-language) • SCIRun (Shared memory, Dataflow, C++) • Uintah (Parallel, Method Invocation, C++) • Kepler (Single process + web services, Generalized dataflow, Java +) • SCIRun2 (Distributed/Parallel, Multi-model, mutli-language)

  3. DOE Common Component Architecture Project • A CA for large-scale Scientific Computation • Component Characteristics • May be SPMD or multi-threaded parallel objects • Heterogeneity • Parallel platforms to desktops and any language • Local and Remote • Parallel communication for remote parallel interfaces and 0-copy in-process connection • Dynamic Composition and Integration • Hot-swapable components, shared instances • www.cca-forum.org • Open forum involving DOE labs, Universities, others

  4. Uintah Problem Specification XML Simulation (One of Arches, ICE, MPM, MPMICE, MPMArches, …) Load Balancer Simulation Controller Callbacks Assignments Tasks Data Archiver Scheduler Tasks Configuration Callbacks MPI • CCA-ish component architecture (C++ only) • Plus components for multiphysics structured AMR simulations • Scales to 2000+ processors

  5. SCIRun

  6. SCIRun PowerApps: BioImage

  7. The KEPLER Systemfor Scientific Workflows … • A framework for design, execution and deployment of scientific workflows • Caters specifically to the domain scientist • Builds on Ptolemy II • Application pull from • various projects http://kepler.ecoinformatics.org Slide thanks to: Ilkay Altintas and Efrat Jeager SDSC UCSD

  8. Kepler Workflow

  9. Component Architecture Design Choices • Degree of isolation: processes, threads, single address space? • Mechanism for communication: dataflow, process networks, method invocation • Synchronization • Programming languages: expressiveness tradeoffs • Data types explicitly supported • Performance requirements • Extra tools required? • Explicit support for parallelism? • Multiple designs for component architectures • Tailored to application needs • Islands of functionality

  10. SCIRun2 • SCIRun2 provides a component model for component models (metacomponents) • Plug-ins provide support for: • CCA • SCIRun • Vtk • Others • Components use “native” communication mechanisms to connect to similar components • Bridges connect models SCIRun2

  11. Meta-components example Common Framework Function Driver Integrator CCA CORBA Bridge Function Driver Integrator

  12. Application

  13. SDM Requirements • Distributed Workflow • Repetitive • Shared resources • Automatically driven • Coarse-grained (seconds to minute per operation) • Interactive Visualization • Exploratory • Dedicated resources • User-driven • Fine-grained (milliseconds to seconds per operation)

  14. Goal • What the user wants • To get work done • Make hard things easy • How to do this • Combine tools with disparate strengths • Make them work efficiently • Focus on interfaces • Enable consistent user interfaces

  15. Utah's Contibution To the SPA Group • SCIRun can now be controlled from SPA/Kepler workflows • Server interface • JNI interface • “Smart” Re-run capability • Provenance framework

  16. Kepler Workflow

  17. Workflow Requirements and “Wants” We Address • Seamless access to resources and services • “Smart” re-runs • Data provenance • Reliability and fault-tolerance • Detached execution • From:B. Ludäscher, et al. Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice & Experience, Special Issue on Scientific Workflows, to appear, 2005.

  18. SCIRun With SPA/Kepler • Kepler actor sends requests to a SCIRun server • Useful for processing batch jobs or iterating through the parameter space of a SCIRun module (actor) • Requires existing SCIRun network, which the workflow actor will tell SCIRun to load • JNI interface to SCIRun

  19. SCIRun Server • Simple TCP/IP server that can be started remotely by Kepler • Accepts requests from client actor in the workflow and then sends back location of results when it has finished • Allows for the possibility of remote or/and detached execution of SCIRun

  20. SCIRun and Kepler Dataflow Integration Incorporate SCIRun computation and visualization with the SPA workflow engine Automate SCIRun network execution with a Kepler actor driving execution through a JNI interface or a remote connection to a SCIRun server

  21. JNI interface with workflow

  22. What is provenance data? • In general: steps taken to get a result • Information about computational experiments or runs of scientific workflows that is needed to reproduce results • We want to log metadata, steps applied to data, tools used to create data products • Useful when you want to share/publish results

  23. The Standalone Provenance Framework http://kepler-project.org/Wiki.jsp?page=KeplerProvenanceFramework

  24. “Smart” re-runs • Instead of running a workflow from scratch we only re-run parts of the workflow that have not been done before • Example: we change a parameter downstream and dont want to re-run the actors that lead up to the one with the parameter change • Especially useful in visualization pipelines and long running workflows

  25. Utah and “Smart” Re-runs • Uses VisTrails’ cache manager algorithm* • Idea is to re-run as little of the network as possible by combining intermediate results from different workflow runs • Recreates input to actors that need to be re-fired • * L. Bavoil, et al. VisTrails: Enabling Interactive Multiple-View Visualizations. IEEE Visualization, 2005.

  26. What is needed for “Smart” Re-runs • We need to keep track of what we have done before • Specifically we need to know what actors have been given what inputs with what outputs • Stored provenance data can give us the information we need

  27. Other uses for provenance data • Recreate results • Recover from a system failure • Checkpoint a workflow • Create semantic links

  28. Future work • Continue work on “Smart” Re-runs system • Help workflow users integrate SCIRun with their workflows • Get provenance framework checked into Ptolemy CVS • Work on other provenance issues • Help SCIRun users take advantage of workflow technology • Develop CCA to Kepler bridging mechanisms

More Related