Grid Computing in the Industry
Download
1 / 57

Grid Computing in the Industry - PowerPoint PPT Presentation


  • 63 Views
  • Uploaded on

Grid Computing in the Industry. North Carolina Statewide Grid. Duke University, Nov 2, 2004. Wolfgang Gentzsch Managing Director MCNC Grid Computing and Networking Services

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Grid Computing in the Industry' - nadine-beck


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Grid Computing in the Industry

North Carolina Statewide Grid

Duke University, Nov 2, 2004

Wolfgang Gentzsch

Managing Director MCNC

Grid Computing and Networking Services

This Lecture aims at complementing your background in distributed computing technologies and developments, providing an actual overview of Grid Computing, and its applications and benefits for Education, Industry and Economy.


My current Background:

MCNC &

The Grid Infrastructure

for North Carolina

NC Statewide Grid Initiative

MCNC Enterprise Grid

  • MCNC

  • Microelectronics Center of North Carolina

  • Founded in 1980

  • Independent, private, non-profit organization

  • Operates NCREN since 1984

  • Past: Supercomputer Center for universities

  • Now: Grid Service Provider, offering Video,

  • Network, Grid, and Datacenter Services

  • 50+ employees

NC BioGrid

NCREN

North Carolina Research & Education Network


The Industry’s View of a Grid :

An IT Utility on a . . .

Grid Middleware (the glue)

Managing . . .

Networked Distributed

Resources


Why Should We Care about Grids ?

“ It’s the next big thing ! “

Grid technologies advance Science and Education in that we can do things which haven’t been possible before.

Grid infrastructure attracts and enables new businesses and creates new jobs, especially in today's rural areas.

Grids make us more competitive by better utilizing resources, bringing results ‘to market’ faster, and delivering with higher quality.


Grid Benefits

Department, Enterprise, and Global Grids

  • Access: transparent, remote, secure, wireless

  • Sharing:enable collaboration over the network

  • Failover: migrate/restart applications automatically

  • On Demand: get resources, when you need them

  • Productivity:more work done in shorter time

  • Virtualization: access compute services, not servers

  • Heterogeneity:platforms, OSs, devices, software

  • Resource Utilization: increase from 20% to 80+%

  • Virtual Organizations: build & dismantle on the fly


However, There is Still a Long Way to Go !

  • Grids are over-hyped: currently, they promise much more

    than they can really offer.

  • Grid technology is far from mature and complete.

  • Grid standards are (mostly) still missing.

  • Grids are very complex IT infrastructures.

  • Grids bring new challenges: sharing resources, loosing

    direct control, security, intellectual property, legal, social,

    political issues . . .


Grid Architecture,

Technology, Standards


Architectural requirements
Architectural Requirements

  • Data Distribution

    • Location, volume, refresh

    • Security of data

  • Usability

    • Administrative Skill set and

    • Client environment

    • Psychological factors important

  • Operations Management

    • 100s CPUs / SA

    • Resources added in large blocks

    • Change control is critical

  • Availability

    • Downtime impact

    • Individual jobs

    • Maintenance windows

  • Scalability

    • Growth 1-3-5 years

    • Scaling strategy and

    • Response to peak loads

    • Technology refresh, evolution

  • Manageability

    • Skill set / workload of SA

    • Expected stability

    • Code management, software distribution

  • Security

    • User authentication

    • Internet access

    • Data Security requirements


The globus toolkit and the open grid services architecture ogsa

... other interfaces

Grid Service

Service

Data

Element

Service

Data

Element

Service

Data

Element

Implementation / Interfaces

Hosting Env/runtime: C, J2EE

Systems Infrastructure

The Globus Toolkit and the Open Grid Services Architecture (OGSA)

  • Integration of Grid technologies and Web Services

  • OGSA defines a “Grid Service”

  • In terms of WSDL interfaces, defines mechanisms:

  • creating distributed systems

  • lifetime management

  • change management

  • credential management

  • notification

Notification

Authorization

Service Creation

Service Registry

Manageability

Concurrency


Global grid forum ggf
Global Grid Forum (GGF)

  • Community-driven set of working groups that are developing standards and best practices for distributed computing ("Grids" and "Metacomputing") efforts

  • Formed in 2001 (merger of US, APAC, Euro efforts)

  • Open Grid Services Architecture (OGSA)


Enterprise grid alliance ega
Enterprise Grid Alliance (EGA)

  • Industry-driven consortium to implement standards in industry products and make them interoperable

  • Founding members: Oracle, Sun, Fujitsu,

  • ‘Enforce’ complementary position in pushing GGF into the academic corner


The changing grid landscape

Sys Mgmt

The Changing Grid Landscape

Developer View:

Write here

Run anywhere

Service Composition

Utility Computing

HPTC Grid

OnDemand

Web Svcs

“Compute Power

from the wall socket”

Standards

Public

SOA

Outsourcing/Hosting

Software as a Svc

. . .

HP

Technologies

IBM

IBM

Enabled Business Models

Enterprise

IT Mgmt

Products

Sun

Market making / Productization

Details see backup slides



Department grids

Internet

Department Grids

Browser to CGM

(Remote Server Setup & Configuration)

Compute Grid

Manager

Auto OS Deployment

Grid Installation/Mgmt

Centralized Server Mgmt

Auto Download of Modules

Solaris

Servers

Linux Servers

Workstations

(Linux or Solaris)


Enterprise grid reference architecture

Gbit-E switch

Gbit-E switch

Gbit-E switch

Gbit-E switch

Gbit-E switch

V240 / V880 NFS

V240 / V880 NFS

V880 QFS/NFS Server

V880 QFS/NFS Server

V240 / V880 NFS

V240 / V880 NFS

FC Switch

Enterprise Grid Reference Architecture

Browser Access

via GEP

SunRay Access

Workstation Access

Optional Control Network (Gbit-E)

Myrinet

Myrinet

Servers,

Blades,

& VIZ

Myrinet

Linux Racks

Grid Manager

Workstations

Sun Fire Link

Data Network (Gbit-E)

NAS/NFS

Simple NFS

HA NFS

Scalable QFS/NFS


Grid enterprise policy management

Project C

Team B

User 1

Department 5

Department 4

Contractor X

User 2

Department 3

Project A

Department 1

Department 2

Department resource access

Campus wide resource demand

Grid Enterprise Policy Management

Clear definition and documentation of priorities

and policies is key to Grid resource management


Globus avaki over multiple local grids

a

Proprietary Data

b

c

d

e

SGE

f

g

h

SGE

SGE

Data mapped and available through Avaki Data Grid

Clients

view

/

Avaki HPC

a

b

c

d

e

f

g

h

Globus & Avaki over multiple local Grids

  • Grid Engine cluster mgmnt within an admin domain & file system area

  • Globus/Avaki knits together resources, handles files, binary management, and high level resource selection

Organization B

Organization C

Organization A

Avaki Data Grid


Applications for the grid
Applications for The Grid

  • Single-CPU Jobs:jobmix, many users, many serial applications, suitable for grid (e.g in universities and research centers)

  • Array Jobs: 100s/1000s of jobs, one user, one serial application, varying input parameters, suitable for grid (e.g. parameter studies in Optimization, CAE, Genomics, Finance)

  • Massively Parallel Jobs:one job, one user, one parallel application, no/low communication, scalable, fine-tune for grid (time-explicit algorithms, film rendering, pattern recognition)

  • Parallel Jobs:one job, one user, one parallel application, high interprocs communication, not suitable for distribution over the grid, but for parallel system in the grid (time-implicit algorithms, direct solvers, large linear algebra equation systems)



Computing as a utility enhancing the grid with a business model
Computing as a Utility =>> Enhancing The Grid with a Business Model

  • On Demand: Get a service at your finger tip

  • From the Wall Socket: Don't care about the infrastructure

  • Metering & Billing: Pay as you go, for what you use

What's a Utility?

Like electricity, water, gas, heat, telephony


Customer challenges

  • Time-to-market Pressures vs. Quality Products

  • Multiple, concurrent projects

  • Projects with different schedules and milestones

  • Overlapping demands for resources

  • Large and growing data sets

  • Larger, more complex product designs

  • Not enough throughput

  • Low utilization of compute resources

  • Insufficient in-house IT expertise

  • Increasingly complex operations management

  • Heightened focus on budget/TCO/ROI

Customer Challenges


Critical customer requirements

  • More Compute Power

  • Improved Resource Access

  • Increased Productivity

  • Better Utilization of Existing Resources

  • Reduced Costs

  • Reduced Management Complexity

Critical Customer Requirements


Who uses department and enterprise grids
Who Uses Department and Enterprise Grids?

  • Life Sciences

    • Startup and cost efficient

    • Custom research or limited use applications

    • Multi-day application runs (BLAST)

    • Exponential Combinations

    • Limited administrative staff

    • Complementary techniques

  • Electronic Design

    • Time to Market

    • Fastest platforms, largest Grids

    • License Management

    • Well established application suite

    • Large legacy investment

    • Platform Ownership issues

  • Financial Services

    • Market simulations

    • Time IS Money

    • Proprietary applications

    • Multiple Platforms

    • Multiple scenario execution

    • Need instant results & analysis tools

  • High Performance Computing

    • Parallel Reservoir Simulations

    • Geophysical Ray Tracing

    • Custom in-house codes

    • Large scale, multi-platform execution


How real is all this stuff
How Real is all this Stuff?

direction of technology adaptation

vendorinterest

researchactivity

department enterprise global


Company Example:

Sun Microsystems


Sun microsystems grid computing
Sun Microsystems & Grid Computing

  • Slogan: The network is the computer

  • Strategy: N1, manage N computers as 1

  • Acquisitions

    • Gridware => Sun Grid Engine, Distributed Resource Manager

    • Pirus => Dynamic Storage Provisioning

    • Terraspring => Automatic System Provisioning

    • Centerrun => Automatic Application Provisioning


Managing grids what we did for the big smps we are now doing for the network
Managing GridsWhat we did for the big SMPs......we are now doing for the Network

  • Resource Virtualization

  • Domains, Zones

  • Interdomain resource mgmt

  • Routing

  • Soft cabling within the box

Solaris N1


Virtualization of resources
Virtualization of Resources

Service1

Vertical

Horizontal

StorageNetwork

Service2

Service3

Compute

Storage

Compute & Data Grid


Services sharing resources

Services

Services

Services

2

3

1

VirtualNetwork

VirtualStore

VirtualCompute

N1: managing services, not servers


N1 provisions grid computing

Grid Compute Service

DB

App

Svr

Web Server

N1 Provisions Grid Computing

DB

App

Svr

App

Svr

Web Server

Web Server

Web Server

App

Svr

Web Server

DB


The n1 effect on efficiency radical improvement in costs uptime
The N1 Effect on EfficiencyRadical improvement in costs & uptime

  • Sys Utilization

80+ %

N1

6-15%

500+

  • Server/Admin

15-30

Costs

100TB

  • Terabytes/DBA

1TB

500+

  • Ports/Admin

50-100

99.999+ %

  • Availability

99.9 %

Uptime

Hours

  • Time to Deploy

Weeks


Grid Computing

Use Cases,

Deployments,

Best Practices


Example: White Rose Grid in England


Wrg hardware
WRG Hardware

Maxima

Snowdon

Pascali

Titania

Maxima


Wrg architecture overview
WRG Architecture Overview

portal

GT2.0

White Rose Grid

GT2.0

GT2.0

GT2.0

GT2.0

GEEE

GEEE

GEEE

GEEE

Solaris

Solaris

Solaris

Linux

Titania

Maxima

Snowdon

Pascali


Wrg key components
WRG Key Components

  • Globus Toolkit 2.4

    Provides a secure means for inter-campus actions

    • Transferring jobs

    • Moving data

    • Gathering information about resources

  • Grid Engine Enterprise Edition

    Manages the campus grid compute resources

    • Delivers a single interface for a heterogeneous grid

    • Guarantees a share of campus resource for grid and local users


  • Wrg key components1
    WRG Key Components

    • Grid Portal Development Kit

      Provides a portal interface into Globus Toolkit

      • Transferring jobs

      • Moving data

      • Gathering information about resources

  • MyProxy

    MyProxy provides a server with client-side utilities to store and retrieve delegated X.509 credentials via the Grid Security Infrastructure (GSI).


  • Grid engine enterprise edition share policies
    Grid Engine Enterprise Edition, Share Policies

    GT2.0

    GT2.0

    GEEE

    GEEE

    Solaris

    Solaris

    GT2.0

    GT2.0

    GEEE

    GEEE

    Solaris

    Linux

    White Rose Grid


    Example:

    Grid Service Provider MCNC & The Grid Infrastructure

    for North Carolina

    NC Statewide Grid Initiative

    MCNC Enterprise Grid

    • MCNC

    • Microelectronics Center of North Carolina

    • Founded in 1980

    • Independent, private, non-profit organization

    • Operates NCREN since 1984

    • Past: Supercomputer Center for universities

    • Now: Grid Service Provider, offering Video,

    • Network, Grid, and Datacenter Services

    • 50+ employees

    NC BioGrid

    NCREN

    North Carolina Research & Education Network


    GCNS Grid Computing and Networking Services

    Mission

    Advance education, innovation and economic development

    throughout North Carolina

    by delivering next generation information technology

    that enables the academic, research, government and

    business communities

    to discover, create, share and apply knowledge.


    The grid the next it infrastructure of north carolina
    The GridThe next IT Infrastructure of North Carolina

    • Proving ground for Grid

    • Successful prototype apps

    • Catalyst for collaboration

    • International recognition

    NC BioGrid

    2002 2003 2004 2005


    The Grid

    The next IT Infrastructure of North Carolina

    • Cluster and SMP resources

    • Research platform for GTEC

    • Core component in NCGrid

    • Revenue generation

    MCNC Enterprise Grid

    NC BioGrid

    2002 2003 2004 2005


    The Grid

    The next IT Infrastructure of North Carolina

    • State-wide partnership

    • Leverage lessons learned

    • Grid education & training resource

    • Enable first mover applications

    NC Grid Initiative

    MCNC Enterprise Grid

    NC BioGrid

    2002 2003 2004 2005


    Mcnc 3 year gcns grid roadmap
    MCNC: 3-Year GCNS Grid Roadmap

    Grid Service Provider:

    • Network, Computing, Data, Video

    • Partner with SC Sites

    • Build GTEC Service Portfolio

    • Start Grid Consulting

    • Annual GSP Workshop

    Easy Access:

    Training, Web Courses

    Access Grid Node

    Grid Appliance

    QuadA

    Awareness Creation:

    Conferences, Workshops, PR

    MCNC Enterprise Grid

    NC BioGrid

    NC Statewide Grid

    CY04

    CY03

    CY05

    CY06


    North carolina s foundation for grid ncren

    Internet Internet 2

    NLR

    Internet

    North Carolina’s Foundation for Grid: NCREN

    4-7 MCNC-owned Clusters

    distributed throughout the state

    Locations still under evaluation

    Cisco

    EPA

    Existing: Blend of owned and leased fiber and circuits moving toward resilient rings powered by Cisco routers

    Planned: Strong focus on owned and leased fiber, Lambda, and few circuits, in resilient rings powered by Cisco routers and Wave Division Multiplexers


    GCNS Enterprise Grid

    Avaki

    Data

    Grid

    Global Grid Resource DB (GIIS)

    Users

    Campus Grids

    Grid Gatekeeper / Interactive Nodes

    Data Grid Access Servers

    (8 total, i.e. 1 per 8 nodes)

    LSF Master Job Scheduler

    32-CPU SGI Altix

    Linux SMP Server

    128-CPU IBM Linux

    Cluster (64 nodes)

    8-TB Storage


    Network grid and data center services
    Network, Grid and Data Center Services

    GTEC, NLR, ANR and other Innovation Initiatives

    DEPLOYMENT

    Self-serve Data Center Services

    Value-add Information Systems Services

    Enterprise Grid Services

    State-wide Grid Services

    Data Archival Services

    Information Security Services

    Hosting & Infrastructure

    Grid Computing

    Information Assurance

    DATA CENTER

    NCREN


    The nc state wide grid roadmap
    The NC State-Wide Grid Roadmap

    • 06/04: “Do-Grid-Yourself” Workshop

    • 07/04: Phase 1, Awareness Creation

    • 10/04 – 06/05: Deliver/Connect Grid Appliance

      Clusters to University Partners in NC

    • 10/04: Develop “Do-Grid-Yourself” Training Course

    • 10/04: Start QuadA Project: Access, Accounting,

      Authentication, Authorization

    • 12/04: Start Deliver Grid Training to Partner Univs

    • 01/05 – 06/05: Work with Grid Users to Port Apps

    • 03/05: Build Access Grid Node

    • 06/05: 1st NC Statewide Grid Workshop


    Grid projects driving grid adoption
    Grid Projects Driving Grid Adoption

    MCNC Grid SP

    Grid ColLab

    Kids Grids

    Gaming Grid

    School Grids

    Grid-Info Grid

    Download-The-Grid

    AAAA Project

    Grid Training Courses

    Startup Grid

    Grid Appliance

    Grid Portal

    GGF & EGA

    NC Statewide Grid

    MCNC Enterprise Grid

    NC BioGrid

    MCNC Supercomputing

    2000 20004 2008



    Our Vision :

    The Three Waves of Grid Computing

    The Research Wave The Industry Wave The Consumer Wave

    Technology, Prototypes Grid-Enabled Products Commodity

    Virtual Organizations Enterprise Solutions IT Utility

    Standards Interoperability Integration

    GGF, IETF, OASIS GGF, EGA, IETF, OASIS Legal, Ethical, Political Orgs

    GCNS: “Awareness Creation” GCNS: “Easy Access” GCNS: “Grid Service Provider”


    Grids Today - Grids in 3 - 5 Years

    Focus on Research - Focus on R&D and business

    Compute-oriented - Petaflops linked w/ Petabytes

    Proprietary interfaces - Standards: GGF, OGSA, DRMAA

    “Mental Firewall” - Security, policies, identity

    Difficult to build - Standards, services, solutions

    Difficult to manage - Sun N1, IBM Autonomic, HP DC

    Difficult to use - Grid Portals: transparent, remote, secure

    Many technologies - Globus Toolkit 3.x


    Security

    Scalability

    Affordability

    Manageability

    NetworkComputer

    Interoperability

    Availability

    Performance

    Mobility—E2E

    Services/Solutions/Partners


    The next 10 years the grid the operating system of the next internet
    The Next 10 Years:The Grid,the Operating System of the next Internet:

    High Capacity: rich in resources

    High Capability: rich in options

    Persistent: stable infrastructure & knowledgeable workforce

    Evolutionary: able to adapt to new technologies & users

    Usable: accessible, robust, and easy-to-use

    Scalable: growth must be part of the design

    Flexible: able to support new applications

    Fault-tolerant: resilient to changes and errors

    Grid Computing: Making the Global Infrastructure a RealityBerman, Fox, Hey, 2003


    Finally anyone anywhere anytime any device any data connected to the grid
    Finally:Anyone, anywhere, anytime, any device, any data,connected to The Grid

    Integration of new devices, data and information sources

    Cell phones, PDAs, smart sensors, sensor arrays, health

    monitors

    Devices embedded in cars, engines, roads, bridges, clothes,...

    Huge amount of data for real-time analysis

    Policies, grid economy, to maintain stability and efficiency

    Organizational and societal structures, to bridge political

    and social boundaries


    Vision grid the time machine of 21 st century
    Vision: Grid, the Time Machine of 21st Century

    • 19th Century Time Machine: Steam Engine

      • Enabled mass production, reduced time to market

    • 20th Century Time Machine: Combustion Engine

      • Much faster from Munich to Dortmund

    • 21st Century Time Machine: Grid Engine

      • Safer, better, new, more products & services & research

      • in much shorter time than today !

    Grid will have great impact on Science, Business, and Society

    ( See examples in the following presentations )


    Time Machines

    The Innovation Engine

    Thank You !

    [email protected]

    http://www.mcnc.org


    ad