SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

1. SHIWA Simulation Platform:Supporting Workflow InteroperabilityTamas Kiss and Gabor Terstyanszky, University of WestminsterEGI Technical ForumPrague17-21 September 2012 SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2. Workflow Interoperability Challenge

3. SHIWA Project Sharing Interoperable Workflows for Large-Scale Scientific Simulation on Available DCIs Partners: Duration: July 2010 – June 2012 (extended to September 2012) 3

4. SHIWA Objectives and Services Objectives: • To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows through a simulation platform • To provide coarse- and fine-grained workflow interoperability solution • To offer Distributed Computing Infrastructure interoperability solution • To provide desktop computer and portal interfaces to manage workflows Key actors: • researchers workflow engine developers workflow developers

5. SHIWA Data Structure • Workflow description abstract workflow • plain text which describes the inputs and outputs and explains what the workflow does • Workflow implementations concrete workflow • It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. • Workflow configurations • It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. • Workflow engines • It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

6. Coarse-Grained Interoperability = CGI Workflows of Workflow Engine A Workflow of Workflow Engine B J1 Submission client Submission Service Workflow Engine B WF2 WF4 Workflow Engine A J3 DCI • native workflows: J1, J3 and WF2 • non-native workflows: WF4 • - black boxes which are managed as legacy code applications

7. CGI Infrastructure DCIs SHIWA Science Gateway SHIWA Repository Unicore DCI ARC DCI SHIWA Portal WF1 WFn GEMLCA admin gLite DCI Globus DCI WF1 WFm WS-PGRADE Workflow editor ASKALON WE Galaxy WE WE1 WEp Triana WE Taverna WE WS-PGRADE Workflow engine GWES WE Kepler WE GEMLCA with GIB Pegasus WE MOTEUR WE GEMLCA Service Proxy Server ProActive WE PGRADE WE SHIWA Proxy Server Workflow Engines • SHIWA Science Gateway Resources native WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.5 WE submission to local cluster repository SHIWA repo remote resources: through remotely pre-deployed submitter GEMLCA with GIB WEs to ARC, gLite, GlobusUnicore DCIs proxy management SHIWA Proxy Server

8. SHIWA Portal: Editing Abstract Workflow Jobs Ports 8

9. SHIWA Portal: Configuring Concrete Workflow

10. SHIWA Portal: Executing Meta-Workflow

11. SHIWA Workflow Repository Table View Browse Workflows Browse Implementations Search Sort

12. SHIWA Workflow Repository Browse View

13. CGI User Scenario DCIs Unicore DCI gLite DCI ARC DCI Globus DCI SHIWA Science Gateway SHIWA Repository ASKALON WE Galaxy WE WF1 WFn Triana WE Taverna WE GWES WE step 1search WF Kepler WE step 3retrieve WF data Pegasus WE WF1 WFm MOTEUR WE SHIWA Portal WE1 WEp step 2 edit WF ProActive WE PGRADE WE GEMLCA Repository WS-PGRADE Workflow editor step 5retrieve WF WE + WF Workflow Engines WF list user WS-PGRADE Workflow engine GEMLCA with GIB step 7 run WF step 6retrieve proxy GEMLCA Service step 4submit WF Proxy Server SHIWA Proxy Server

14. Fine-Grained Interoperability = FGI Workflow of Workflow Engine B Workflow of Workflow Engine A J1 J1 J2 J3 J2 J3 J4 J4 WFA ->IWIR frontend plug-in IWIR -> WFB backend plug-in Workflow Engine B J1 J1 J2 J2 J3 J3 DCI J4 J4 IWIR workflow IWIR workflow • front-end plug-in: converts workflow of workflow engine A into IWIR (Intermediate Workflow Interpreter Representation) • back-end plug-in: converts from IWIR into workflow of workflow engine B 14 14

15. FGI Scenario: Manual Mode step 1search WF SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn step 2download WF(A) Front-end Plug-ins c step 3upload IWIR WF FP1 FP1 FP1 WS-PGRADE Workflow Editor WS-PGRADE Workflow Engine step 4download IWIR WF c BP1 BP1 BP1 Back-end Plug-ins step 5upload WF(B) 15

16. FGI Scenario: Automatic Mode SHIWA Science Gateway step 1search WF SHIWA Portal SHIWA Repository WF1 WFn step 2edit WF step 3retrieve WF GEMLCA Repository WF1 WFm Workflow Editor WE1 WFp c WS-PGRADE Workflow Editor FP1 BP1 Workflow Engine FP2 BP2 c WS-PGRADE Workflow Engine DCI Bridge JSDL Translator FPn BPm step 5translate WF step 6 translate WF Meta Broker Unicore DCI ARC DCI Proxy Server gLite DCI Globus DCI DCIs 16 16

17. workflow for DCI B DCI Interoperability jobs in non-JSDL J1 J2 J3 jobs in JSDL J4 J1 Workflow Engine JSDL Translator J2 J3 J4 DCI Bridge DCI Metabroker Proxy Server 17

18. DCI Bridge • Features: • web service based app which provides standard access to different DCIs: • clouds - EC2, OpenNebula • clusters - LSF, PSB • desktop grids - BOINC, OurGrid, XtremWeb • service grids – ARC, gLite, Globus, Unicore • workflow jobs are submitted to the BES Factory Service of the DCI Bridge where jobs are described in JSDL • Components: • resource registry - configuration interface of DCIs • application manager - BES management • runtime system - job execution management via BES Factory • monitoring - logs and messages handling and visualisation

19. DCI Interoperability Scenario SHIWA Science Gateway SHIWA Portal SHIWA Repository WF1 WFn step 3retrieve WF data GEMLCA Repository step 1search WF WF1 WFm step 2 edit WF WE1 WFp WS-PGRADE Workflow editor DCIs step 4 create JSDL Unicore DCI ARC DCI WS-PGRADE Workflow engine DCI Bridge JSDL Translator step 7 run WF gLite DCI Globus DCI step 5 select DCI Meta Broker step 6 retrieve certificate Proxy Server 19

20. SHIWA Simulation Platform: Access • SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator • DCI resources • certificate given by the relevant DCI administrator, for example the VO sysadmin • Access to the SHIWA Portal • ssp.shiwa-workflow.eu • Access to the SHIWA Repository • repo.shiwa-workflow.eu 20

21. SHIWA Simulation Platform: Achievements • Supported workflow systems • ASKALON, Galaxy, GWES, Kepler, MOTEUR, PGRADE, Pro-Active, Taverna, Triana • Pegasus (under development) • Number of workflows • 72 abstract workflows + 98 concrete workflows • Research communities • Life Sciences, Neuroscience, Medical Imaging,… • Number of workflow execution • around 1260 and increasing • Number of registered users • 65 users 21 21

22. Conclusions • the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows • users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities • workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform • researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform • the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources

23. Thank you for your attention … Any questions?