1 / 36

ICENI Overview & Grid Scheduling

ICENI Overview & Grid Scheduling. Laurie Young London e-Science Centre Department of Computing, Imperial College. ICENI. IC e -Science N etworked I nfrastructure Developed by LeSC Grid Middleware Group Collect and provide relevant Grid meta-data

giovanna-abbott
Download Presentation

ICENI Overview & Grid Scheduling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ICENI Overview&Grid Scheduling Laurie Young London e-Science Centre Department of Computing, Imperial College

  2. ICENI ICe-Science Networked Infrastructure Developed by LeSC Grid Middleware Group Collect and provide relevant Grid meta-data Use to define and develop higher-level services Interaction with other frameworks: OGSA, Jxta etc. The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.

  3. ICENI (The Big Picture) Web Services Gateway Public Computational Community Computational Resource Application Portal CR SR Identity Manager JavaCoG Globus Private Administrative Domain Storage Resources Domain Manager CR CR Resource Browser Public Computational Community SR SR Network Resources SR Software Resources Application Mapper Policy Manager Resource Application CR Broker Component Design Tools Design Tools SR Resource Manager Gateway between private Public Private and public regions

  4. ICENI Stack Portal Interface Application Construction & Deployment ICENI Middleware Grid Fabric

  5. Web Portals • Handheld wireless devices become ubiquitous • Personal Digital Assistants, Mobile Phones • Secure access any time, any place, any where • Use X.509 certificates embedded in a browser to authenticate user’s identity • Integration portal infrastructure with ICENI • EPIC: Use component meta-data to build portal application • Goal: Provide secure ‘one stop shop’ for e-science

  6. EPIC: e-Science Portal at Imperial College • Collaborative LeSC industrial project with Sun Microsystems • Develop a secure portal infrastructure to: • Access your own personal environment • Applications to support day-to-day e-science • Interaction with other Grid infrastructures • Allow role based access to resources • Anonymous: public web pages • Students: internal pages, email, compute resources • Staff: restricted pages

  7. ICENI Application Model • Legacy code! • Component Applications • Compose applications from many components • Component does work on data • Component communicates data

  8. Component Motivation • Logical application model • Collaborative software authoring • Promote component reuse and sharing • Simplify application construction • Enable deployment to diverse Grid resources: • Communication Selection • Implementation Selection

  9. Layered Abstraction Meaning dataflow abstract data types may have many Behaviour Behaviour control flow threads etc. may each have many performance, architectures, concrete data type Implementation Implementation

  10. Component’s View of the Grid Context object Other Code More Code SOAP RMI You must implement a provided interface You may call methods provided by the middleware My Code

  11. Visual Component Composition

  12. Deployment of Components Access Resource Information Application Description Document Application Mapper Repository Run-Time Representation Code APO Code Code Code Grid Container Code Application Proxy Object Component Design Tools Implementation Annotating Tools Scientist Developer Application Design Tools End User RTR

  13. Component Execution OGSA, Jxta, etc. OGSA, Jxta, etc. Jini Jini MPI SOAP APO Code Code Code RTR RTR RTR RMI Network Resource Compute Resource Hardware

  14. Components as Services Component Service interface Context object SOAP (or other) protocol

  15. ICENI & Jini: P2P

  16. Web Services Architecture

  17. Synergy

  18. Grid Service Contracts Jini Lookup Service DRMAA Client DRMAA Resource

  19. Grid Service Contracts Jini Lookup Service Resource Browser DRMAA Client DRMAA Resource User: A+B Duration:1hr User:B

  20. Grid Service Contracts Jini Lookup Service User:A Duration:10m User:A DRMAA Client DRMAA Resource DRMAA Client DRMAA Resource User: A+B Duration:1hr User:B

  21. OGSA & Jini Integration Jini Lookup Service GSI enabled Web Service Hosting Environment User:A Duration:10m Gateway Manager DRMAA Resource DRMAA Resource User: A+B Duration:1hr

  22. OGSA & Jini Integration Jini Lookup Service User:A Duration:10m GSI enabled Web Service Hosting Environment Gateway Manager DRMAA Resource WSDL Interface DRMAA Resource User: A+B Duration:1hr Jini Client Interface

  23. OGSA & Jini Integration Jini Lookup Service User:A Duration:10m GSI enabled Web Service Hosting Environment Gateway Manager DRMAA Resource WSDL Interface DRMAA Resource GSI + SOAP Connection User: A+B Duration:1hr Jini Client Interface

  24. OGSA & Jini Integration Jini Lookup Service User:A Duration:10m GSI enabled Web Service Hosting Environment Gateway Manager DRMAA Resource WSDL Interface SOAP->Java DRMAA Resource GSI + SOAP Connection User: A+B Duration:1hr User Info Jini Client Interface

  25. Application Mapping (Scheduling) • Architecture • How meta-data is collected • What meta-data is required • Scheduling Algorithms • Map components onto resources for “best” results • Meta-data dependent decisions

  26. Scheduling Architecture ICENI App Builder (GUI) Component Repository Performance Models Scheduler Launcher Resources

  27. Multiple Metrics (1) • “It is the goal of a scheduler to optimise one or more metrics” (Feitelson & Rudolph) • Generally one metric is most important • Application Optimisation • Execution time • Execution cost • Host Optimisation • Host utilisation • Host throughput • Interaction Latency

  28. Multiple Metrics (2) • In a Grid Environment there are three application optimisation based important metrics • Start time ( ) • End time ( ) • Cost ( ) • Relative importance varies on a user by user and application by application basis

  29. Combining Metrics – Benefit Fn • A Benefit Function maps the metrics we are interested in to a single Benefit Value metric • Different benefit functions represent different optimisation preferences

  30. Optimisation Preferences • Cost Optimisation • Time Optimisation • Cost/Time Optimisation

  31. Schedule Benefit • Each component and communication has a benefit function • Each resource and network connection has a predicted time & cost for each component or communication that could be deployed • Fit the tasks onto the resources to get the maximum Total Predicted Benefit

  32. Graph Oriented Scheduling (1) • Applications are described as a graph • Nodes represent application components • Edges represent component communication • Resources are described as a graph • Nodes represent resources • Edges represent network connections

  33. Graph Oriented Scheduling (2) Atlas Saturn Design Factory Analyse Scatter Viking Mesh Mesh Mesh Condor pool DRACS DRACS DRACS Gather

  34. Graph Oriented Scheduling (3) Analyse Factory Gather Scatter Atlas Saturn Design Condor pool Viking

  35. Summary • Component framework provides: • Rich application meta-data • Decoupled component definition and implementation • Meta-Data: • Exploit performance information to map component implementation to the ‘best’ resources • Resource Broker: • Resource selection through user defined policies: • Minimise cost using computational economics • Minimise execution time using the application mapper

  36. Acknowledgements • Director: Professor John Darlington • Technical Director: Dr Steven Newhouse • Research Staff: • Anthony Mayer, Nathalie Furmento • Stephen McGough, James Stanton • Yong Xie, William Lee • Marko Krznaric, Murtaza Gulamali • Asif Saleem, Laurie Young, Gary Kong • Contact: • http://www.lesc.ic.ac.uk/ • e-mail: lesc@ic.ac.uk

More Related