CoG & Eclipse

1. U.S. Department of Energy Office of Science CoG & Eclipse Gregor von Laszewski Argonne National Laboratory University of Chicago gregor@mcs.anl.gov http://www.mcs.anl.gov/~gregor Sponsors:

2. Outline • What is the Cog Kit? • Selected Highlights • Eclipse integration • Conclusion Motivation Analysis Implementation Conclusion

3. What is the CoG Kit?

4. Abstractions • Hypothesis: • With rapidly changing technologies it may be beneficial to have an abstraction that can be assisting in this technical challenge of building scientific infrastructures using the Grid. • Solution • The CoG Kit provides abstractions that are version independent and allow easy integration in applications. A dynamic workflow framework is provided.

5. Key Concepts • CoG Pattern • Abstractions • Task abstractions for elementary Grid operations • Tasks to include commodity tools and frameworks • Task Event Model • Tasks = jobs, transfers, authentication, … , … • Providers • Integration of different Grid and non Grid middleware • Workflows • Queues, Sets of tasks • Hierarchical Graphs of Tasks • Dynamic workflow framework • Visuals Abstractions • Gridfaces (Abstractions for visual components) • Deployment independence • Change on the fly (e.g. during run)

6. The Integrated Approach Standalone Applications Desktop Web Portals Java CoG Kit Increased exposed Usability Command line tools Abstractions Services APIs Increased exposed Functionality

7. Development Support Architecture: CoG Abstraction Layers Nano materials Bio- Informatics Disaster Management Portals Applications Gridfaces Layer Data and Task Management Layer GridIDE CoG Abstraction Layer CoG CoG CoG CoG CoG CoG CoG GT2 GT3 OGSA/ WS-RF SSH Condor Unicore Others Avaki Others SETI

8. Partial UML diagram

9. Defining a CoG/Task public void statusChanged (StatusEvent event) { Status status = event.getStatus( ); logger.debug(``Status changed to '' + status.getStatusCode()); if (status.getStatusCode( ) == Status.COMPLETED) { logger.info(``Task Done''); elsif (status.getStatusCode( ) == Status.FAILED) { logger.info(``Task Failed''); System.exit(1); } } public class COG implements StatusListener{ public void create() { … } public void submit () { … } public void statusChanged (StatusEvent e) { … } public static void main (String arg[]){ try { COG cog = new COG(); cog.create(); cog.submit(); catch (Exception e) { logger.error(“Something went wrong:”, e); } }

10. Defining Workflows try { handler.submit (cog); } catch (InvalidSecurityContextException ise) { logger.error(``Security Exception'', ise); System.exit(1); } catch (TaskSubmissionException tse) { logger.error(``TaskSubmission Exception'', tse); System.exit(1); } catch (IllegalSpecException ispe) { logger.error(``Specification Exception'', ispe); System.exit(1); } catch (InvalidServiceContactException isce){ logger.error(``Service Contact Exception'', isce); System.exit(1); } Task 3 TaskGraph tg = new TaskGraphImpl(); public void create () { // define tasks ….. /* Add the tasks to the TaskGraph */ tg.add(task1); tg.add(task2); tg.add(task3); tg.add(task4); tg.addDependency(task1, task2); tg.addDependency(task1, task3); tg.addDependency(task2, task4); tg.addDependency(task3, task4); } public void submit() { TaskGraphHandler handler = new TaskGraphHandlerImpl(); try { handler.submit(tg); } catch (Exception e) { logger.error(``Some Error occured'', e); System.exit(1); } } Task 1 Task 4 Task 2

11. Defining Tasks Task task4 = new Task(); JobSpecification spec = new JobSpecificationImpl(); spec.setExecutable(“/share/climate”); spec.addArguments(“-dimension 512 512”); spec.setStdOutput(“output.txt”); task4.setSpecification(spec); Task task3 = new Task(); JobSpecification spec = new JobSpecificationImpl(); spec.setExecutable(“/share/climate”); spec.addArguments(“-dimension 512 512”); spec.setStdOutput(“output.txt”); task3.setSpecification(spec); Task task2 = new Task(); JobSpecification spec = new JobSpecificationImpl(); spec.setExecutable(“/share/climate”); spec.addArguments(“-dimension 512 512”); spec.setStdOutput(“output.txt”); task2.setSpecification(spec); Task task1 = new Task(); JobSpecification spec = new JobSpecificationImpl(); spec.setExecutable(“/share/bin/climate”); spec.addArguments(“-dimension 512 512 512 –task 1”); spec.setStdOutput(“output1.txt”); task1.setSpecification(spec); Task 4 Task 3 Task 2 Task 1 Provider 4 Provider 3 Provider 2 Provider 1

12. Assigning a Service to Work on a Task Service service = new ServiceImpl(Service.JOB_SUBMISSION); service.setProvider(``GT3_2_1''); // Set Security Context – e.g. certificates and such SecurityContext securityContext = CoreFactory.newSecurityContext(``GT3_2_1''); securityContext.setCredentials(null); // e.g. set it to default in ./globus service.setSecurityContext(securityContext); // Set Contact – e.g. where to go to ServiceContact serviceContact = new ServiceContactImpl( http://hot.mcs.anl.gov:8080/ogsa/services/base/gram/, MasterForkManagedJobFactoryService”); service.setServiceContact(serviceContact); tas2.setService(Service.JOB_SUBMISSION_SERVICE, service); Service service = new ServiceImpl(Service.JOB_SUBMISSION); service.setProvider(``GT3_2_1''); // Set Security Context – e.g. certificates and such SecurityContext securityContext = CoreFactory.newSecurityContext(``GT3_2_1''); securityContext.setCredentials(null); // e.g. set it to default in ./globus service.setSecurityContext(securityContext); // Set Contact – e.g. where to go to ServiceContact serviceContact = new ServiceContactImpl( http://hot.mcs.anl.gov:8080/ogsa/services/base/gram/, MasterForkManagedJobFactoryService”); service.setServiceContact(serviceContact); tas2.setService(Service.JOB_SUBMISSION_SERVICE, service); Service service = new ServiceImpl(Service.JOB_SUBMISSION); service.setProvider(``GT3_2_1''); // Set Security Context – e.g. certificates and such SecurityContext securityContext = CoreFactory.newSecurityContext(``GT3_2_1''); securityContext.setCredentials(null); // e.g. set it to default in ./globus service.setSecurityContext(securityContext); // Set Contact – e.g. where to go to ServiceContact serviceContact = new ServiceContactImpl( http://hot.mcs.anl.gov:8080/ogsa/services/base/gram/, MasterForkManagedJobFactoryService”); service.setServiceContact(serviceContact); tas2.setService(Service.JOB_SUBMISSION_SERVICE, service); Service service = new ServiceImpl(Service.JOB_SUBMISSION); service.setProvider(``GT3_2_1''); // Set Security Context – e.g. certificates and such SecurityContext securityContext = CoreFactory.newSecurityContext(``GT3_2_1''); securityContext.setCredentials(null); // e.g. set it to default in ./globus service.setSecurityContext(securityContext); // Set Contact – e.g. where to go to ServiceContact serviceContact = new ServiceContactImpl( http://hot.mcs.anl.gov:8080/ogsa/services/base/gram/, MasterForkManagedJobFactoryService”); service.setServiceContact(serviceContact); task1.setService(Service.JOB_SUBMISSION_SERVICE, service);

13. And here is the XML workflow <project> <include file="sysdefaults.xml"/> <sequential> <task:execute executable="/share/bin/climate" arguments="-dimension 512 512 512 -task 1" host="hot.mcs.anl.gov" provider="gt3.2.1" stdout="output1.txt"/> <parallel> <task:execute executable="/share/bin/climate" arguments="-dimension 512 512 512 -task 2" host="cold.mcs.anl.gov" provider="gt2" stdout="output2.txt"/> <task:execute executable="/share/bin/climate" arguments="-dimension 512 512 512 -task 3" host="hot.mcs.anl.gov" provider="gt3.2.1" stdout="output3.txt"/> </parallel> <task:execute executable="/share/bin/climate" arguments="-dimension 512 512 512 -task 4“ host="blue.mcs.anl.gov" provider="gt4" stdout="output4.txt"/> </sequential> </project>

14. A Functional Language project( include("sysdefaults.xml“) seq( task:execute("/share/bin/climate", "-dimension 512 512 512 -task 1“, "hot.mcs.anl.gov“, "gt3.2.1“, "output1.txt“) par( task:execute("/share/bin/climate", "-dimension 512 512 512 -task 2“, “cold.mcs.anl.gov“, "gt2“, "output2.txt“) task:execute("/share/bin/climate", "-dimension 512 512 512 -task 3“, "hot.mcs.anl.gov“, "gt4“, "output3.txt“) ) task:execute("/share/bin/climate", "-dimension 512 512 512 -task 4“, "hot.mcs.anl.gov“, "gt4“, "output4.txt“) ) )

15. Visual Abstractions Un-submitted System Icons Submitted Active Icons Toolbar Active Grid File Browser and Grid File Transfer Completed Failed Active Icons change dynamically their appearance depending on the state of the calculation. Grid Monitor Native Icons

16. Hierarchical Graph Visualization

17. Architecture View: Abstractions CoG CoG CoG GT3.02 GT3.02 GT3.02 Nano materials Bio- Informatics Disaster Management Portals Applications Workflow Applications Workflow Design & Monitoring Gridfaces Layer Workflow Portlet Workflow Vizualizer/Editor Data and Task Management Layer Karajan Workflow Frameworks GridAnt CoG Kit Abstraction Layer Workflow Abstractions Workflow Abstractions Queue/Set Abstractions CoG CoG CoG CoG CoG CoG CoG Kit providers SSH GT4 WS-RF local WebDAV GT2 Grid Middleware & Commodity Technologies

18. Architecture View: Components Programming Tools Specification Simple Workflow Language Workflow Abstractions Editor Workflow XML Workflow APIs Monitor Workflow Services Debugger Workflow Portlets Command line tools Translators Instantiation and Execution Workflow Engine Grid

19. Integration into Eclipse • Provide project files that directly load into eclipse • Provide a separate download that is nightly updated for a streamlined version • Develop GUIs that use SWT • Develop a workflow GUI in SWT • … • Provide simplified GUIs that work without the desktop • Develop a “sidebar” with components that can be called • Expose high level tools such as workflows, job queues and parameter studies • Develop tools that simplify the integration with GT4

20. Project files • Java CoG Kit modules • Dependencies and properties • Automatic generation of .project and .classpath files • A multi-project import would help

21. Visual Workflow Editor • Use Eclipse GEF to design CoG workflows • Either: • Karajan workflows • Plain Java workflows (using the Abstractions API) • Editing • Live monitoring

22. IDE for Karajan • Syntax highlight (both XML and native) • Debugger integration • Visualization • Semantic editing

23. Further Information • URL • http://www.cogkits.org/ • Contact • gregor@mcs.anl.gov