Grid Execution Management for Legacy Code Applications

1. Grid Enabling Legacy Code Applications GridExecutionManagementforLegacyCodeApplications Tamas Kiss Centre for Parallel Computing University of Westminster kisst@wmin.ac.uk

2. Legacy Applications • Code from the past, maintained because it works • Often supports business critical functions • Not Grid enabled What to do with legacy codes when utilising the Grid? • Bin them and implement Grid enabled applications • Reengineer them • Port them onto the Grid with minimum user effort

3. GEMLCA – Grid Execution Management for Legacy Code Architecture Objectives • To deploy legacy code applications as Grid services without reengineering the original code and minimal user effort • To create complex Grid workflows where components are legacy code applications • To make these functions available from a Grid Portal GEMLCA GEMLCA PGPortal Integration

4. GEMLCA Concept Invoke legacy code Send custom input Compute Servers Client 2: to invoke legacy code Grid service GEMLCAGrid service Return result Produce result • Resource managerdeploys: • LC • GEMLCA Client 1: to registerlegacy code as a Grid service

5. The GEMLCA-view of a legacy code • Any code that correspond to the following model can be exposed as Grid service by GEMLCA: Legacy code Input data (files, command line params, env. vars) Output data (files) . . . Perform computation Call shared libraries Communicate over the network Query databases

6. GEMLCA Concept Invoke legacy code Custom input Compute Servers Client 2: to invoke legacy code Grid service GEMLCAGrid service Return result Produce result • Resource managerdeploys: • LC • GEMLCA Client 1: to registerlegacy code as a Grid service Code is deployed statically Legacycode Input data is provided dynamically Access to the code is controlled by the GEMLCA Grid service

7. Implementing the concept • The GEMLCA service can be implemented with any grid/service-oriented technology E.g: • Globus 3 or 4 currently available implementations • Jini • Web services • … • GEMLCA service could invoke legacy codes in many different ways. Current implementation: • Submit the legacy code as a batch job to a local jobmanager through a Grid middleware layer (GT3/4)

8. What’s behind the GEMLCA service… • Consequences: • Not only the legacy code and the GEMLCA service, but also a local jobmanager and a Grid middleware layer must be installed on the hosting system! • The aim of the code registration process is to tell GEMLCA how to submit the legacy code to the grid middleware layer Compute GEMLCA Service Gridmiddlewarelayer Client Servers Grid Service Client GEMLCA Job submission and file transfer services Process Legacy Local jobmanager Code Job e.g. Condor/Fork

9. What’s the point? • Heterogeneous codes can be hidden behind the same interface (the programming interface of the GEMLCA service) • Different programs can be invoked in the same way • Extend non grid-aware programs with security infrastructure (access enabled through a Grid service) • Share your codes with your colleagues or partner institutes • Expose business logic to your employers or customers • Build customized GEMLCA clients (such as the GEMLCA P-GRADE Portal) • Compose complex processes by connecting multiple legacy code grid services together

10. The GEMLCA P-GRADE Portal A Web-based GEMLCA client environment… University of Westminster, LondonMTA SZTAKI, Budapest

11. The aim of the GEMLCA P-GRADE Portal • To provide graphical clients to GEMLCA with a portal-based solution • To enable the integration of legacy code grid services into workflows

12. 3rd generation Grids: (OGSA: GT4, gLite) Web browser Grid Site 1 LC 1 Result Desktop 1 Legacy applications Grid Site 2 Web browser LC 2 Input Legacy applications Desktop N . . . The GEMLCA P-GRADE Portal GEMLCAP-GRADE Portal Server

13. The GEMLCA-specific version of the P-GRADE Portal is different from the original P-GRADE Portal! • It contains a web page to register legacy codes as grid services • It contains a GEMLCA-specific workflow editor • Workflow components can be “legacy code grid services” (not only batch jobs) • It contains a GEMLCA-specific workflow manager subsystem • It can invoke GEMLCA services (not only submitting jobs)

14. Legacy code registration page “GEMLCA Administration Tool” portlet

15. Legacy code registration page

16. GEMLCA workflow editor in a nutshell Workflow Creation UoW Sztaki UoR

17. User provides the binary executable User provides input data for the executable Job produces result files Executable is already installed on a machine User provides input data for the executable Job produces result files Batch components vs. GEMLCA components in P-GRADE Portal workflows Batch component GEMLCA component Workflow components must be defined in different ways Input files represented by ports Output files represented by ports Ports guarantee compatibility  batch and GEMLCA components can mutually produce data to each other!

18. Combine legacy codes with new codesinside the same workflow! Combining legacy and non-legacy components Code invocation Code invocation Job submission Code invocation Job submission

19. Defining GEMLCA components Workflow Creation UoW Sztaki UoR

20. An example for a GEMLCA workflow Workflow to analyse road traffic Manhattan road network generator Traffic simulators Analyser

21. I’m a biologist not a Grid expert Sorry, no additional utilities to be deployed on core resources ??? UK NGS Leeds Oxford • Service reliability • Administration GT2 Manchester Rutherford GEMLCA and Production Grids The problem

22. User-friendly Web interface UK NGS Leeds Oxford GT2 Manchester Portal server at UoW Rutherford The solution

23. 3rd generation Grids: (OGSA: GT3, WSRF, gLite) Web browser Grid Site 1 Desktop 1 Legacy applications Grid Site 2 Web browser Legacy applications Desktop N Centralised GEMLCA solution P-GRADE Portal Server

24. interface GEMLCA interface GEMLCA legacy code repository 3rd party service provider (UoW) GEMLCA resource (GT4 + GEMLCA classes) NGS site1 (GT2) Workflow definition NGS site2 (GT2) P-Grade Portal Central repository legacy code1 legacy code2 …. legacy coden NGS siten (GT2) user job submission

25. Advantages of the GEMLCA legacy code repository • legacy codes can be uploaded into a central repository and made available for authorised users through a Grid portal • extends the usability of NGS as users utilise others’ legacy codes stored in the repository • No support needed at the NGS sites

26. The P-GRADE NGS GEMLCA Portal • Portal Website:http://www.cpc.wmin.ac.uk/ngsportal/ • How to get access? • Obtain a UK e-Science Certificatehttp://www.grid-support.ac.uk/ • Obtain an NGS user accounthttp://www.ngs.ac.uk/ • Obtain an P-GRADE NGS portal accounthttp://www.cpc.wmin.ac.uk/ngsportal/

27. Conclusions • GEMLCA enables the deployment of legacy code applications as Grid services without any real user effort. • GEMLCA is integrated with the P-GRADE portal to offer user-friendly development and execution environment. • The integrated GEMLCA P-GRADE solution is available for the UK NGS as a service! • www.cpc.wmin.ac.uk/ngsportal

28. Further GEMLCA Developments • Extending existing Grid brokers and production Grid systems with resource availability prediction based on historical data.

29. Further GEMLCA Developments Automatic service deployment, discovery and selection Grid Site 1 Web browser Information Service P-GRADE Portal Server Desktop 1 Broker Legacy applications Grid Site 2 Web browser Abstract workflow Automatic deployer Desktop N Legacy applications

30. Live demonstration of the P-GRADE GEMLCA portal

31. Any questions? Thank you for your attention … Email: gemlca-discuss@cpc.wmin.ac.uk Websites: www.cpc.wmin.ac.uk/gemlca www.lpds.sztaki.hu/pgportal www.cpc.wmin.ac.uk/ngsportal