The cc grid era cc gs c 2002
This presentation is the property of its rightful owner.
Sponsored Links
1 / 19

The CC – GRID? Era CC GS C 2002 PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

The CC – GRID? Era CC GS C 2002. Gordon Bell ([email protected]) Bay Area Research Center Microsoft Corporation. Observations from a mostly Grid workshop. Clusters. Let’s finish the job! Grids generally. Grids as arbitrary cluster platforms…why?

Download Presentation

The CC – GRID? Era CC GS C 2002

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

The cc grid era cc gs c 2002

The CC – GRID? EraCCGSC 2002

Gordon Bell ([email protected])

Bay Area Research Center

Microsoft Corporation

Observations from a mostly grid workshop

Observations from a mostly Grid workshop

  • Clusters. Let’s finish the job!

  • Grids generally.

  • Grids as arbitrary cluster platforms…why?

  • Examples of Grid-types, especially web services

  • Summary…

Blades aka a cluster in a cabinet

Blades aka a “cluster in a cabinet”

  • 366 servers per 44U cabinet

    • Single processor

    • 2 - 30 GB/computer (24 TBytes)

    • 2 - 100 Mbps Ethernets

  • ~10x perf*, power, disk, I/O per cabinet

  • ~3x price/perf

  • Network services… Linux based

    *42, 2 processors, 84 Ethernet, 3 TBytes

The cc grid era cc gs c 2002

Clusters aren’t as bad as programs make them out to be, but we need to make them work better and be more transparent.

  • Everything is becoming a cluster. Certainly all of 500!

  • 64 bit addressing will cause more change!

  • Future nodes should bet on CLMP smP’s (p = 4-32) .Utilize existing and emerging smP’s nodes versus assuming lcd PM-pairs & MPI.

  • Massive gains from compiler and runtime. ES has set a new standard of efficiency and system transparency for “clusters”.

  • Expand the MPI programming model:

    • Full transparency of MPI needs to be the goal

    • Objectify for greater flexibility and greater insulation from latency

Grids if they are the solution what s the problem

Grids: If they are the solution what’s the problem?

  • Economics… thief, scavenger, power, efficiency or resource sharing?

  • Research funding… that’s where the money is

  • Are they where the problems lie?

  • Does massive collaboration that the Grids enable, create massive overhead and generally less output?Unless the output is for a community!

  • Is funding and middleware a good investment?

Same observations as 2000

Same observations as 2000


  • GRID was/is an exciting concept …

    • They can/must work within a community, organization, or project. Apps need to drive.

    • “Necessity is the mother of invention.”

  • Taxonomy… interesting vs necessity

    • Cycle scavenging and object evaluation (e.g. [email protected], QCD)

    • File distribution/sharing for IP theft e.g. Napster

    • Databases &/or programs for a community(astronomy, bioinformatics, CERN, NCAR)

    • Workbenches: web workflow chem, bio…

    • Exchanges… many sites operating together

    • Single, large objectified pipeline… e.g. NASA.

    • Grid as a cluster platform! Transparent & arbitrary access including load balancing

Web SVCs

Grid n j an arbitrary distributed cluster platform

Grid nj. An arbitrary distributed, cluster platform

A geographical and multi-organizational collection of diverse computers dynamically configured as cluster platforms responding to arbitrary, ill-defined jobs “thrown” at it.

  • Costs are not necessarily favorable e.g. disks are less expensive than cost to transfer data.

  • Latency and bandwidth are non-deterministic, thereby changing cluster characteristics

  • Once a large body of data exists for a job, it is inherently bound to (set into) fixed resources.

  • Large datasets & I/O bound programs need to be with their data or be database accesses…

  • But are there resources there to share?

  • Bound to cost more?

Bright spots near term user focus a lesson for grid suppliers

Bright spots… near term, user focus, a lesson for Grid suppliers

  • Tony Hey apps-based funding. Web services based Grid & data orientation.

  • David Abramson - Nimrod.

    • Parameter scans… other low hanging fruit

    • Encapsulate apps! “Excel”-- language/control mgmt.

    • “Legacy apps are programs that users just want, and there’s no time or resources to modify code …independent of age, author, or language e.g. Java.”

  • Andrew Grimshaw - Avaki

    • Making Legion vision real. A reality check.

  • Lip 4 pairs of “web services” based apps

  • Gray et al Skyservice and Terraservice

  • Goal: providing a web service must be as easy as publishing a web page…and will occur!!!

The cc grid era cc gs c 2002

SkyServer: delivering a web service to the astronomy community. Prototype for other sciences? Gray, Szalay, et al

First paper on the SkyServer

Later, more detailed paper for database community

What can be learned from sky server

What can be learned from Sky Server?

  • It’s about data, not about harvesting flops

  • 1-2 hr. query programs versus 1 wk programs based on grep

  • 10 minute runs versus 3 day compute & searches

  • Database viewpoint. 100x speed-ups

    • Avoid costly re-computation and searches

    • Use indices and PARALLEL I/O. Read / Write >>1.

    • Parallelism is automatic, transparent, and just depends on the number of computers/disks.

  • Limited experience and talent to use dbases.

Heuristics for building communities that need to share data programs

Heuristics for building communities that need to share data & programs

  • Always go from working to working

  • Do it by induction in time and space(Why version 3 is pretty good.)

  • Put ONE database in place that’s useful by itself in terms of UI, content, & queries

  • Invent and demo 10-20 instances of use

  • Get two working in a single location

  • Extend to include a second community, with an appropriate superset capability

Some science is hitting a wall ftp and grep are not adequate jim gray

You can GREP 1 GB in a minute

You can GREP 1 TB in 2 days

You can GREP 1 PB in 3 years.

1PB ~10,000 >> 1,000 disks

At some point you need indices to limit searchparallel data search and analysis

Goal using dbases. Make it easy to

Publish: Record structured data

Find data anywhere in the network

Get the subset you need!

Explore datasets interactively

Database becomes the file system!!!

You can FTP 1 MB in 1 sec.

You can FTP 1 GB / min.

… 2 days and 1K$

… 3 years and 1M$

Some science is hitting a wallFTP and GREP are not adequate (Jim Gray)

Network concerns

Network concerns

  • Very high cost

    • $(1 + 1) / GByte to send on the net; Fedex and 160 GByte shipments are cheaper

    • DSL at home is $0.15 - $0.30

  • Disks cost less than $2/GByte to purchase

  • Low availability of fast links (last mile problem)

    • Labs & universities have DS3 links at most, and they are very expensive

    • Traffic: Instant messaging, music stealing

  • Performance at desktop is poor

    • 1- 10 Mbps; very poor communication links

  • Manage: trade-in fast links for cheap links!!

Gray s 2 4 k 1 tbyte sneakernet aka disk brick

Gray’s $2.4 K, 1 TByte Sneakernet aka Disk Brick

Cost to move a Terabyte

Cost, time, and speed to move a Terabyte

Cost of a “Sneaker-Net” TB

  • We now ship NTFS/SQL disks.

  • Not good format for Linux.

  • Ship NFS/CIFS/ODBC servers (not disks).

  • Plug “disk” into LAN.

    • DHCP then file or DB serve…

    • Web Service in long term

Courtesy of Jim Gray, Microsoft Bay Area Research

Cost to move a terabyte

Cost to move a Terabyte

Cost time of sneaker net vs alts

Cost, time of Sneaker-net vs Alts

Courtesy of Jim Gray, Microsoft Bay Area Research

Grids real and personal two carrots one downside a bet

Grids: Real and “personal”Two carrots, one downside. A bet.

  • Bell will match any Gordon Bell Prize (parallelism, performance, or performance/cost) winner’s prize that is based on “Grid Platform Technology”.

  • I will bet any individual or set of individuals of the Grid Research community up to $5,000 that a Grid application will not win the above by SC2005.

The cc grid era cc gs c 2002

The EndHow can GRIDs become a real, useful, computer structure?Get a life. Adopt an application community!Success if CCGSC2004 is the last…by making Grids ubiquitous.

  • Login