1 / 33

Concepts of grid computing

Concepts of grid computing. Guy Warner gcw@nesc.ac.uk. Acknowledgements. This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by: Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre) the EDG training team

tass
Download Presentation

Concepts of grid computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Concepts of grid computing Guy Warner gcw@nesc.ac.uk

  2. Acknowledgements • This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by: • Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre) • the EDG training team • Ian Foster, Argonne National Laboratories • Jeffrey Grethe, SDSC • EGEE colleagues • Mark Baker, The Distributed Systems Group, University of Portsmouth, http://dsg.port.ac.uk/mab • Talks at 3rd EGEE conference by • Kyriakos Baxevanidis,Deputy Head,Unit of Research Infrastructures,European Commission, DG INFSO • Dr Spyros Konidaris, European Commission – DG INFSO Concepts of Grid Computing, Towards e-Research, St Andrews

  3. Goals of this module • To introduce the concepts of Grid computing assuming no previous knowledge Concepts of Grid Computing, Towards e-Research, St Andrews

  4. Contents • “The Grid” vision • What is “a grid” ? • Drivers of grid computing • Current status of grids • The basis: authentication, authorisation, security Concepts of Grid Computing, Towards e-Research, St Andrews

  5. The Grid Metaphor Mobile Access G R I D M I D D L E W A R E Supercomputer, PC-Cluster Workstation Data-storage, Sensors, Experiments Visualising Internet, networks Concepts of Grid Computing, Towards e-Research, St Andrews

  6. The grid vision • The grid vision is of “Virtual computing” (+ information services to locate computation, storage resources) • Compare: The web: “virtual documents” (+ search engine to locate them) • MOTIVATION: collaboration through sharing resources (and expertise) to expand horizons of • Research • Commerce – engineering, … “the knowledge economy” • Public service – health, environment,… Concepts of Grid Computing, Towards e-Research, St Andrews

  7. Contents • “The Grid” vision • What is “a grid” ? Concepts of Grid Computing, Towards e-Research, St Andrews

  8. Institute A Institute C Institute B Institute D “A grid” • The initial vision: “The Grid” • The present reality: Many “grids” • Each grid is an infrastructure enabling one or more “virtual organisations” to share computing resources • What’s a VO? • People in different organisations seeking to cooperate and share resources across their organisational boundaries • Why establish a Grid? • Share data • Pool computers • Collaborate VO Concepts of Grid Computing, Towards e-Research, St Andrews

  9. Application Software Operating System Disks, Processor, Memory, … The Single Computer • The Operating System enables easy use of • Input devices • Processor • Disks • Display • Any other attached devices Concepts of Grid Computing, Towards e-Research, St Andrews

  10. Application Software Middlewarefor sharing computers, servers, printers, … Operating System on each computer Resources connected by a LAN Resources on a Local Area Network User just perceives “shared resources”, with no regard to location in the organisation: - Authenticated by username / password - Authorised to use own files,… Concepts of Grid Computing, Towards e-Research, St Andrews

  11. Resources on a grid Application Software Interface between app. and grid Grid Middleware: “collective services” Grid Middleware on each resource Operating System on each resource Resources connected by internet Concepts of Grid Computing, Towards e-Research, St Andrews

  12. INTERNET A grid • Grid middleware runs on each shared resource • Data storage • (Usually) batch jobs on pools of processors • Users join VO’s • Virtual organisation negotiates with sites to agree access to resources • Distributed services (both people and middleware) enable the grid Concepts of Grid Computing, Towards e-Research, St Andrews

  13. What characterises a grid? • Co-ordinated resource sharing • No centralised point of control • Different administrative domains. • Standard, open, general-purpose protocols and interfaces • NOT specific to an application • EGEE, NGS support multiple VO’s • Delivering non-trivial qualities of service • Co-ordinated to deliver combined services, greater than sum of the individual components • http://www.gridtoday.com/02/0722/100136.html Concepts of Grid Computing, Towards e-Research, St Andrews

  14. The components of a Grid • Resources • networking, computers, storage, data, instruments, … • Grid Middleware • the “operating system of the grid” • Operations infrastructure • Run enabling services (people + software) • Virtual Organization management • Procedures for gaining access to resources Concepts of Grid Computing, Towards e-Research, St Andrews

  15. Key concepts • Virtual organisation: people and resources collaborating - across admin, organisational boundaries • Single sign-on • I connect to one machine – some sort of “digital credential” is passed on to any other resource I use, basis of: • Authentication: How do I identify myself to a resource without username/password for each resource I use? • Authorisation: what can I do? Determined by • My membership of VO • VO negotiations with resource providers • Grid middleware runs on each resource • User just perceives “shared resources” with no concern for location or owning organisation Concepts of Grid Computing, Towards e-Research, St Andrews

  16. Contents • “The Grid” vision • What is “a grid” ? • Drivers of grid computing Concepts of Grid Computing, Towards e-Research, St Andrews

  17. The first driver: e-Science • What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise...) • Often very compute intensive • Often very data intensive (both creating new data and accessing very large data collections) – data deluges from new technologies • Crosses organisational boundaries • Examples…. Concepts of Grid Computing, Towards e-Research, St Andrews

  18. Astronomy No. & sizes of data sets as of mid-2002, grouped by wavelength • 12 waveband coverage of large areas of the sky • Total about 200 TB data • Doubling every 12 months • Largest catalogues near 1B objects Data and images courtesy Alex Szalay, John Hopkins University Concepts of Grid Computing, Towards e-Research, St Andrews

  19. Large Hadron Collider at CERN • Data Challenge: • 10Petabytes/year of data !!! • 20 million CDs each year! • Simulation, reconstruction, analysis: • LHC data handling requires computing power equivalent to ~100,000 of today's fastest PC processors! • Operational challenges • Reliable and scalable through project lifetime of decades Mont Blanc (4810 m) Downtown Geneva Concepts of Grid Computing, Towards e-Research, St Andrews

  20. dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf BLAST dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf Seq1 > dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf BLAST Seq1 > dcscdssdcsdcdsc bscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfd Seq2 > bvdfvfdvhbdfvb bhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkc … Seqn > bvdfvfdvhbdfvb bhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkchdsqhfduhdhdhqedezhhezldhezhfehflezfzejfv dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf DB dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf DB dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf Seq2 > dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf Seqn > dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf BLAST dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf DB dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf RESULT dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfdbvdfvfdvhbdfvbhdbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkcqhdsqhfduhdhdhqedezhdhezldhezhfehflezfzeflehfhezfhehfezhflezhflhfhfelhfehflzlhfzdjazslzdhfhfdfezhfehfizhflqfhduhsdslchlkchudcscscdscdscdscsddzdzeqvnvqvnq! Vqlvkndlkvnldwdfbwdfbdbd wdfbfbndblnblkdnblkdbdfbwfdbfn BLAST dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf DB BLAST gridification Computing element Input file UI Computing element Concepts of Grid Computing, Towards e-Research, St Andrews

  21. DAME: Grid based tools and Infer-structure for Aero-Engine Diagnosis and Prognosis Engine flight data London Airport Airline office New York Airport Grid Diagnostics Centre Maintenance Centre American data center European data center “A Significant factor in the success of the Rolls-Royce campaign to power the Boeing 7E7 with the Trent 1000 was the emphasis on the new aftermarket support service for the engines provided via DS&S. Boeing personnel were shown DAME as an example of the new ways of gathering and processing the large amounts of data that could be retrieved from an advanced aircraft such as the 7E7, and they were very impressed”, DS&S 2004 XTO Companies: Rolls-Royce DS&S Cybula Universities: York, Leeds, Sheffield, Oxford Engine Model Case Based Reasoning Concepts of Grid Computing, Towards e-Research, St Andrews

  22. Political drivers • Entering the “knowledge society” from the “industrial society” • industrial society: also enabled by communications infrastructure • Lisbon strategy: Research and Innovation will be the most important factors in determining Europe’s success through the next decades • THE GOAL: “UNLEASH CREATIVITY”- by investment in • Human skills • Infrastructures • Growth of e-infrastructure (= networks + grid + operations) • phase 1: mainly academia, some in industry: “an elite, privileged to do this job” • phase 2: ordinary people doing distributed work; SMEs, adopt, adapt and use • phase 3: the next generations • will transform e-infrastructure and its uses • We don’t know how others will use what we devise Concepts of Grid Computing, Towards e-Research, St Andrews

  23. Contents • “The Grid” vision • What is “a grid” ? • Drivers of grid computing • Some examples • Current status of grids Concepts of Grid Computing, Towards e-Research, St Andrews

  24. If “The Grid” vision leads us here… … then where are we now? Concepts of Grid Computing, Towards e-Research, St Andrews

  25. Grid projects Many Grid development efforts — all over the world UK – OGSA-DAI, RealityGrid, GeoDise, Comb-e-Chem, DiscoveryNet, DAME, AstroGrid, GridPP, MyGrid, GOLD, eDiamond, Integrative Biology, … Netherlands – VLAM, PolderGrid Germany – UNICORE, Grid proposal France – Grid funding approved Italy – INFN Grid Eire – Grid proposals Switzerland - Network/Grid proposal Hungary – DemoGrid, Grid proposal Norway, Sweden - NorduGrid NASA Information Power Grid DOE Science Grid NSF National Virtual Observatory NSF GriPhyN DOE Particle Physics Data Grid NSF TeraGrid DOE ASCI Grid DOE Earth Systems Grid DARPA CoABS Grid NEESGrid DOH BIRN NSF iVDGL DataGrid (CERN, ...) EuroGrid (Unicore) DataTag (CERN,…) Astrophysical Virtual Observatory GRIP (Globus/Unicore) GRIA (Industrial applications) GridLab (Cactus Toolkit) CrossGrid (Infrastructure Components) EGSO (Solar Physics) Concepts of Grid Computing, Towards e-Research, St Andrews

  26. Grids: where are we now? • Many key concepts identified and known • Many grid projects have tested, and benefit from, these • Major efforts now on establishing: • Standards (a slow process) (e.g. Global Grid Forum, http://www.gridforum.org/ ) • Production Grids for multiple VO’s • “Production” = Reliable, sustainable, with commitments to quality of service • In Europe, EGEE • In UK, National Grid Service • In US, Teragrid • One stack of middleware that serves many research (and other!!!) communities • Operational procedures and services (people!, policy,..) • New user communities • … whilst research & development continues Concepts of Grid Computing, Towards e-Research, St Andrews

  27. The tools, services used by the VO’s applications Application development environment, portals, semantics Insulate applications from changing middleware Application Application toolkits, standards Middleware: “collective services” Basic Grid services:AA, job submission, info, … The key for new VO’s Concepts of Grid Computing, Towards e-Research, St Andrews

  28. The vision of 2001: convergence of Web Services and Grids Open Grid Services Architecture web developments “big Science” research OGSIGrid prototypes Web services World-wide web INTERNET High-end computing High throughput-computing Massively parallel computing Concepts of Grid Computing, Towards e-Research, St Andrews

  29. Contents • “The Grid” vision • What is “a grid” ? • Drivers of grid computing • Current status of grids • The basis: authentication, authorisation, security Concepts of Grid Computing, Towards e-Research, St Andrews

  30. Grid security and trust -1 • Providers of resources (computers, databases,..) need risks to be controlled: they are asked to trust users they do not know • They trust a VO • The VO trusts its users • User’s need • single sign-on: to be able to logon to a machine that can pass the user’s identity to other resources • To trust owners of the resources they are using • Build middleware on layer providing: • Authentication: who wants to use/provide resource • Authorisation: what the user is allowed to do • Security: reduce vulnerability, e.g. from outside the firewall • Non-repudiation: knowing who did what • Digital credentials and the “Grid Security Infrastructure” middleware are the basis of production grids Concepts of Grid Computing, Towards e-Research, St Andrews

  31. Grid security and trust -2 • Currently, achieved by Certification: • User’s identity has to be certified by one of the national Certification Authorities (CAs) • mutually recognized http://www.gridpma.org/, for EU go via here to http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.htmlto find your CA • E.g. In UK go to http://www.grid-support.ac.uk/ca/ralist.htm • Resources are also certified by CAs • User • User joins a VO • Digital certificate is basis of AA • Identity passed to other resources you use, where it is mapped to a local account – the mapping is maintained by the VO • Common agreed policies establish rights for a Virtual Organization to use resources Concepts of Grid Computing, Towards e-Research, St Andrews

  32. Grid security and trust -3 • Certification and GSI provides • Authentication • Resource can trust user • User can trust the resource provider • …. So long as certificates are protected – they are your grid identity • A basis for Authorisation • so a VO can manage access to resources • Resource providers trust the VO • The VO trusts the user • Mechanism for checking message integrity • Messages are passed between machines • Public/private key pairs protect message integrity as well as authentication • Not (usually) encrypted but message-integrity is checked Concepts of Grid Computing, Towards e-Research, St Andrews

  33. Summary of grid computing concepts • Flexible collaboration across multiple administrative domains – sharing data, computers, instruments, application software,.. • Single sign-on to resources in multiple organisations • Authorisation, authentication • Need for people-services as well as middleware services • credential authorities, VO managers, support • Drives are towards • Production services (reliable, sustainable,… – against which research projects can plan with confidence) • In Europe, EGEE • In UK, National Grid Service • Standards • Empowering new user communities Concepts of Grid Computing, Towards e-Research, St Andrews

More Related