1 / 29

SCI-BUS: A Generic Science Gateway Framework for EGI User Communities

SCI-BUS is a generic science gateway framework that allows user communities to access multiple distributed computing infrastructures (DCIs) in a transparent way. It provides a scalable architecture for developing and sharing scientific applications across grids, clouds, and clusters.

weigel
Download Presentation

SCI-BUS: A Generic Science Gateway Framework for EGI User Communities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WS-PGRADE/gUSE generic DCI gateway framework for EGI user communities http://www.sci-bus.eu Zoltan Farkas and Peter Kacsuk MTA SZTAKI SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481

  2. Motivations • There are many user communities who would like to access several DCIs (grids, clouds, clusters) in a transparent way • They do not want to learn the peculiar features of the used DCIs • They want to concentrate on their scientific application • Therefore they need a science gateway

  3. Who are the members of an e-science community? • End-users (e-scientists) (50.000-500.000) • Execute the published WF applications with custom input parameters by creating application instances using the published WF applications as templates • Science Gateway (SG) Framework Developers (5-10) • Develop genericSG framework • Writes technical, user and installation manuals • SG Instance Developers (50-100) • Develop application domain specific SG instance • Writes technical, user and installation manuals • WF Application Developers (500-1.000) • Develop WF applications • Publish the completed WF applications for end-users

  4. Using a SG frameworkto develop applications and share them with others Supercomputer based SGs (DEISA, TeraGrid) Access to a large set of ready-to-run scientific WF applications • Accessing a large set of various DCIs to make these WF applications run WF App. Repository SG framework Cluster based service grids (SGs) (EGEE, OSG, etc.) Desktop grids (DGs) (BOINC, Condor, etc.) Grid systems Local clusters Supercomputers E-science infrastructure What do WF developers need? Clouds

  5. Supercomputer based SGs (DEISA, TeraGrid) App. Repository SG instance Cluster based service grids (SGs) (EGEE, OSG, etc.) Desktop grids (DGs) (BOINC, Condor, etc.) Grid systems Local clusters Supercomputers What do e-scientists need? E-scientists • Sharing ready-to-use appls via the repository • Joint run appls via the gateway in the hidden e-science infrastructure Clouds E-science infrastructure

  6. How to build a science gateway? • Option 1: Build from scratch • If the gateway is not extremely simple, it requires long time to develop a robust gateway • Requires substantial manpower and development cost • It is very specialized and as users start to use it and come up with new requirements it is difficult to extend in a scalable way • It typically represents an isolated development without belonging to an open source community

  7. How to build a science gateway? • Option 2: Adapt and customize an existing gateway framework • Significantly reduces development time • Requires limited manpower and development cost • Produces a robust and usable service • The open source community is driving force for further development and extensions SCI-BUS provides the required core gateway and customization technology

  8. Criteria to select an existing gateway framework • Robustness • Sustainability • Will it exist in 3 years time? • How big and trustable the community who develops it? • Functionalities • How easy to adapt for the needs of the new user community? • Scalability • Extendibility

  9. Main objectives of SCI-BUS • Create a generic-purpose gateway framework • Elaborate a gateway instance development technology based on a customisation methodology • Establish production SG instance services both for national grids (horizontal user communities) and various science communities (vertical user communities) • Develop business models to guarantee sustainability and commercial exploitation

  10. Generic-purpose gateway framework • Based on Liferay • WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment) • gUSE(Grid User Support Environment) architecture • General purpose • Workflow-oriented portal framework • Supports the development and execution of workflow-based applications • Enables the multi-DCI execution of any WF • Support the fast development of SG instances by a customization technology

  11. Scalable architecture based on collaborating services

  12. Seamless access to various types of DCIs WEB-UI(HTML) WS (Axis) submitter BOINC submitter Unicore submitter Glite submitter GT5 submitter Cloud submitter ARC submitter DCIs Client machine Portal Server machine BES interface WF Interpreter WF Storage Liferay WS-PGRADE portal ARC Grid DCI-Bridge GT5 Grid File Storage WF Graph editor BOINC Grid Application Repository Cloud Broker Information System

  13. User access modes WS-PGRADE workflow UI WS-PGRADE UI ASM API Customized UI Other, existing UI

  14. Creating and running WS-PGRADE workflow in WF developer view Step 1: Edit workflow

  15. Step 2: Configuring the workflow

  16. Running workflow instance

  17. App. Repositoryy Gateway WS-PGRADE/gUSE community aspects E-scientists Application developers Support collaboration between e-scientists and application developers • Application Developers • Develop e-science applications via thegatewayin collaboration with e-scientists • Publish the completed applications for end-users via an application repository • End-users (e-scientists) • Specify the problem/application needs • Execute the published applications via thegatewaywith custom input parameters by creating application instances

  18. WS-PGRADE/gUSE community aspects • Separated views: • application developer view: (access to all the features of WS-PGRADE/gUSE • end-user view: limited access defined by the gateway sys admin • Built-in workflow/application repository • App developer can upload and download: • Workflows • Templates • Applications • End-user can only download: • Applications

  19. End-user UI: To build SG instances • WF editing (and many other features) are not supported, only: • Appl (WF) download • Appl Parameterization • Appl Submission • Progress monitoring • Result download • E.g. Autodock gateway

  20. Example for SG instance based on end-user view:Autodock gateway

  21. User access modes WS-PGRADE workflow UI WS-PGRADE UI ASM API Customized UI Other, existing UI

  22. Concept of Application Specific Module Client Webbrowser Developed by ASM WS-PGRADE Customized web-interface Internal components gUSE Placing web-app as gUSE component submitter Desktop or Service grids / cloud / Clusters DCIs

  23. Types of gateways • Generic purpose gateway for national grids • Core WS-PGRADE/gUSE (e.g. Greek NGI) • Generic purpose gateway for a particular DCI • EDGI gateway based on WS-PGRADE/gUSE • Generic purpose gateway for specific technologies • SHIWA gateway based on WS-PGRADE/gUSE • Application-oriented science gateway instance

  24. Application-specific SG instances in SCI-BUS • Create 11 SG instances in the 1st project year for the following user communities: • International seismology community • Helio-physics communityc • Swiss systems biology community of the SystemsX.ch project • German MoSGrid computational chemistry and bioinformatics community • Biomedical researchers community of the Academic Medical Centre of the University of Amsterdam • Astrophysics community • PireGrid SMEs community • Business process modelling community • Blender rendering community • Citizen web-2 community • Public application developer community (based on ETICS-2 results) • Create another 6 user community SG instances based on 6 subcontractors • Create another 10 user community SG instances based on associated membership

  25. ETH Proteomics Gateway • Using WS-PGRADE for workflow creation • gUSE for workflow management and submission • Heavy use of the ASM API to present a simplified interface to end-users (this works extremely well) • First instance: Private Gateway for one of the largest Proteomics research groups in Europe (inside ETH): • Up to 100 users • Access to 1200 cores local cluster and to ARC based Swiss Grid • Planned future instance: Public Gateway for high-end Proteomics

  26. RecentWS-PGRADE releases History since v3.4.0 • Nov 2011: v3.4.0 (DCI Bridge) • Feb 2012: v3.4.1 (usage statistics portlet) • March 2012: v3.4.2 (support for new EMI release) • April 2012: v3.4.3 (support for Liferay 6.1) • May 2012: v3.4.4 (EDGI DCI support) • June 2012: v3.4.5 (integrated SHIWA repository) • June 2012: v3.4.6 (installation and config. wizard) • … • Aug 2012: v3.5.0 (SaaS cloud access via CBP) • Sep 2012: v3.5.1 (IaaS cloud access via CBP)

  27. Downloads statistics at sourceforge From 48 countries

  28. Where to find further information? • gUSE/WS-PGRADE: • http://www.guse.hu/ • gUSE on sourceforge • http://sourceforge.net/projects/guse/ • http://sourceforge.net/projects/guse/forums/forum/ • http://sourceforge.net/projects/guse/develop • SCI-BUS web page: • http://www.sci-bus.eu/ • SHIWA web page: • http://www.shiwa-workflow.eu/

  29. Conclusions • If you want to develop a SG instance, use a SG framework instead of developing it from scratch • Why to select WS-PGRADE/gUSE and join the SCI-BUS community? • Robustness • Already large number of gateways used in production • Sustainability • The SCI-BUS project and its sustainability and commercialization plan guarantees it • Functionalities • Rich functionalities that are growing according to the SCI-BUS and sourceforge community needs • How easy to adapt for the needs of the new user community? • Already large number of gateways customized from gUSE/WS-PGRADE • You can influence the progress of WS-PGRADE/gUSE Join SCI-BUS as associated member

More Related