SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

1. : Sharing Workflows through Coarse-Grained Workflow InteroperabilityG. Terstyanszky, T. Kukla, T. Kiss, S. Winter, J.:Centre for Parallel ComputingUniversity of Westminster, London, United KingdomP. Kacsuk, A. BalaskoMTA-SZTAKI, Budapest Hungary SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585

2. Sharing Interoperable Workflows through Large-Scale Scientific Simulation on Available DCIs (SHIWA) Partners: Duration: July 2010 – June 2012 (extended to September 2012) 2

3. Workflow Interoperability Challenge

4. SHIWA Objectives and Services Objectives: • To create an environment which enables seamless execution of workflows of different workflow systems through workflow interoperability Services: • To support the whole workflow lifecycle: editing, uploading, browsing downloading and executing workflows • To provide coarse- and fine-grained workflow interoperability solution • To offer Distributed Computing Infrastructure interoperability solution • To provide desktop computer and portal interfaces to manage workflows Key actors: • researchers workflow engine developers workflow developers

5. WF Developer E-Scientist WE Developer SHIWA Solution Browse Run Upload Upload Developer Portal Admin Portal End User Portal SHIWA WERepository SHIWA WF Repository SHIWA WF enactor fMRI Triana WF description WF engine binary MOTEUR Sequencing WF executables IWIR front,back end Pegasus WF engine XML description MRI simulator NonLinSyst simulator ASKALON Master WF System P-GRADE CGI …. FGI … Source WF System Source WF System Target WF System Run DCI a Embedded WF System 1 Embedded WF System 1 Embedded WF System 2 Export IWIR Export IWIR Import IWIR Run DCI c Run DCI b Run DCI b IWIR 5 5 SHIWA Science Gateway

6. SHIWA Data Structure • Workflow description abstract workflow • plain text which describes the inputs and outputs and explains what the workflow does • Workflow implementations concrete workflow • It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine. • Workflow configurations • It contains pre-defined input files and parameters or references to these files and parameters and other data/metadata of a workflow. • Workflow engines • It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references to them

7. Coarse-Grained Interoperability: Single Non-native Workflow Workflow of Workflow Engine B Submission client Workflow Engine A Submission Service Workflow Engine B WF DCI

8. Coarse-Grained Interoperability: Meta-Workflow Workflow of Workflow Engine A Workflow of Workflow Engine B J1 GEMLCA Submission Service GEMLCA Client Workflow Engine B J2 WF4 Workflow Engine A J3 DCI Resources • native workflows: J1, J2, J3 • non-native workflows: WF4 • - black boxes which are managed as legacy code applications

9. SSP: CGI Infrastructure DCIs SHIWA Science Gateway SHIWA Repository Unicore DCI ARC DCI SHIWA Portal WF1 WFn GEMLCA admin gLite DCI Globus DCI GEMLCA Repository WS-PGRADE Workflow editor WF1 WFm ASKALON WE Galaxy WE Triana WE WE1 WEp Taverna WE WS-PGRADE Workflow engine GWES WE Kepler WE GEMLCA with GIB Pegasus WE MOTEUR WE GEMLCA Service Proxy Server ProActive WE PGRADE WE SHIWA Proxy Server Workflow Engines • SHIWA Science Gateway Resources native WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.1 WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployed submitter GEMLCA with GIB WEs to ARC, gLite, GlobusUnicore DCIs proxy management SHIWA Proxy Server

10. SHIWA Portal: Abstract Workflow Jobs Ports 10

11. SHIWA Portal: Configuring Workflow

12. SHIWA Portal: Executing Workflow

13. SHIWA Repository: Browse View search domain

14. SHIWA Repository: Browse View inputs outputs data sets implementation

15. SHIWA Repository: Table View Browse Workflows Browse Implementations Advanced Search Search Sort 15

16. SHIWA Repository: Execution Support Execution data 16

17. SHIWA Portal: Legacy Code Explorer Taverna workflow

18. SHIWA Portal: Repository Portlet Workflow Repository

19. CGI Developer Scenario: Specifying WE

20. CGI Developer Scenario: Specifying WF through portal

21. CGI Developer Scenario: Specifying WF through desktop

22. CGI User Scenario: Native WE DCIs Unicore DCI gLite DCI ARC DCI Globus DCI SHIWA Science Gateway SHIWA Repository ASKALON WE Galaxy WE WF1 WFn Triana WE Taverna WE GWES WE step 1search WF Kepler WE step 3retrieve WF data Pegasus WE WF1 WFm MOTEUR WE SHIWA Portal WE1 WEp step 2 edit WF ProActive WE PGRADE WE GEMLCA Repository WS-PGRADE Workflow editor step 5retrieve WF WE + WF Workflow Engines WF list user WS-PGRADE Workflow engine GEMLCA with GIB step 7 run WF step 6retrieve proxy GEMLCA Service step 4submit WF Proxy Server SHIWA Proxy Server 22

23. CGI User Scenario: Non-native WE DCIs Unicore DCI gLite DCI ARC DCI Globus DCI SHIWA Science Gateway SHIWA Repository ASKALON WE Galaxy WE WF1 WFn Triana WE Taverna WE GWES WE Kepler WE step 1 search WF GEMLCA Repository Pegasus WE WF1 WFm MOTEUR WE SHIWA Portal WE1 WEp ProActive WE step 5 retrieve WF PGRADE WE WS-PGRADE Workflow Editor user step 2 edit WF WE + WF Workflow Engines GEMLCA with GIB WS-PGRADE Workflow Engine step 4 forward WF step 7 run WF MOTEUR Workflow editor step 3 submit WF GEMLCA Service step 6 retrieve proxy MOTEURWorkflow Engine Proxy Server GEMLCA Client GEMLCA UI SHIWA Proxy Server

24. CGI User Scenario: WF Execution via Desktop step 1 search WF step 4edit WF SHIWA Science Gateway SHIWA Repository step 2 retrieve WF user WF1 WFn SHUWA Desktop step 5ask run WF GEMLCA Repository step 3forward WF bundle WF1 WFm SHIWA Portal WE WE1 WEp step 6submit WF WS-PGRADE Workflow Editor DCIs GEMLCA with GIB WS-PGRADE Workflow Engine Unicore DCI ARC DCI GEMLCA Service gLite DCI Globus DCI Proxy Server SHIWA Proxy Server

25. SHIWA Simulation Platform: Access • SHIWA Science Gateway (SHIWA Portal + SHIWA Repository) • joint portal and repository account given by the science gateway administrator • DCI resources • certificate given by the relevant DCI administrator, for example the VO sysadmin • Access to the SHIWA Portal • ssp.shiwa-workflow.eu • Access to the SHIWA Repository • repo.shiwa-workflow.eu 25

26. Conclusions • the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows • users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities • workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform • researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform • the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources