1 / 50

Building Peta-Byte Servers

Building Peta-Byte Servers. Jim Gray Microsoft Research Gray@Microsoft.com http://www.Research.Microsoft.com/~Gray/talks. Kilo 10 3 Mega 10 6 Giga 10 9 Tera 10 12 today, we are here Peta 10 15 Exa 10 18. Outline. The challenge: Building GIANT data stores

tfeliciano
Download Presentation

Building Peta-Byte Servers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building Peta-Byte Servers Jim Gray Microsoft Research Gray@Microsoft.com http://www.Research.Microsoft.com/~Gray/talks Kilo 103 Mega 106 Giga 109 Tera 1012 today, we are here Peta 1015 Exa 1018

  2. Outline • The challenge: Building GIANT data stores • for example, the EOS/DIS 15 PB system • Conclusion 1 • Think about Maps and SCANS • Conclusion 2: • Think about Clusters

  3. The Challenge -- EOS/DIS • Antarctica is melting -- 77% of fresh water liberated • sea level rises 70 meters • Chico & Memphis are beach-front property • New York, Washington, SF, SB, LA, London, Paris  • Let’s study it! Mission to Planet Earth • EOS: Earth Observing System (17B$ => 10B$) • 50 instruments on 10 satellites 1997-2001 • Landsat (added later) • EOS DIS: Data Information System: • 3-5 MB/s raw, 30-50 MB/s cooked. • 4 TB/day, • 15 PB by year 2007

  4. The Process Flow • Data arrives and is pre-processed. • instrument data is calibrated, gridded averaged • Geophysical data is derived • Users ask for stored data OR to analyze and combine data. • Can make the pull-push split dynamically Push Processing Pull Processing Other Data

  5. Designing EOS/DIS (for success) • Expect that millions will use the system (online)Three user categories: • NASA 500 -- funded by NASA to do science • Global Change 10 k - other dirt bags • Internet 20 m - everyone else Grain speculators Environmental Impact Reports school kids New applications => discovery & access must be automatic • Allow anyone to set up a peer- node(DAAC & SCF) • Design for Ad Hoc queries, Not Just Standard Data Products If push is 90%, then 10% of data is read (on average). => A failure: no one uses the data, in DSS, push is 1% or less. => computation demand is enormous(pull:push is 100: 1)

  6. The (UC alternative) Architecture • 2+N data center design • Scaleable DBMS to manage the data • Emphasize Pull vs Push processing • Storage hierarchy • Data Pump • Just in time acquisition

  7. 2+N Data Center Design • Duplex the archive (for fault tolerance) • Let anyone build an extract (the +N) • Partition data by time and by space (store 2 or 4 ways). • Each partition is a free-standing DBMS (similar to Tandem, Teradata designs). • Clients and Partitions interact via standard protocols • DCOM/CORBA, OLE-DB, HTTP,… • Use the (Next Generation) Internet

  8. Obvious Point: EOS/DIS will be a Cluster of SMPs • It needs 16 PB storage = 1 M disks in current technology = 500K tapes in current technology • It needs 100 TeraOps of processing = 100K processors (current technology) and ~ 100 Terabytes of DRAM • 1997 requirements are 1000x smaller • smaller data rate • almost no re-processing work

  9. Hardware Architecture • 2 Huge Data Centers • Each has 50 to 1,000 nodes in a cluster • Each node has about 25…250 TB of storage (FY00 prices) • SMP .5Bips to 50 Bips 20K$ • DRAM 50GB to 1 TB 50K$ • 100 disks 2.3 TB to 230 TB 200K$ • 10 tape robots 50 TB to 500 TB 100K$ • 2 Interconnects 1GBps to 100 GBps 20K$ • Node costs 500K$ • Data Center costs 25M$ (capital cost)

  10. Scaleable DBMS • Adopt cluster approach (Tandem, Teradata, VMScluster,..) • System must scale to many processors, disks, links • Organize data as a Database, not a collection of files • SQL rather than FTP as the metaphor • add object types unique to EOS/DIS (Object Relational DB) • DBMS based on standard object model • CORBA or DCOM (not vendor specific) • Grow by adding components • System must be self-managing

  11. Storage Hierarchy 10-TB RAM 500 nodes 1 PB of Disk 10,000 drives 15 PB of Tape Robot 4x1,000 robots • Cache hot 10% (1.5 PB) on disk. • Keep cold 90% on near-line tape. • Remember recent results on speculation| research challenge: how trade push +store vs. pull. • (more on this later Maps & SCANS)

  12. Data Pump • Some queries require reading ALL the data (for reprocessing) • Each Data Center scans the data every 2 days. • Data rate 10 PB/day = 10 TB/node/day = 120 MB/s • Compute on demand small jobs • less than 1,000 tape mounts • less than 100 M disk accesses • less than 100 TeraOps. • (less than 30 minute response time) • For BIG JOBS scan entire 15PB database • Queries (and extracts) “snoop” this data pump.

  13. Just-in-time acquisition 30% 5 10 4 10 3 10 2 10 10 1 • Hardware prices decline 20%-40%/year • So buy at last moment • Buy best product that day: commodity • Depreciate over 3 years so that facility is fresh. • (after 3 years, cost is 23% of original). 60% decline peaks at 10M$ EOS DIS Disk Storage Size and Cost assume 40% price decline/year Data Need TB Storage Cost M$ 1994 1996 1998 2000 2002 2004 2006 2008

  14. Just-in-time acquisition 50%!!!!!!! • Hardware prices decline 50%/year lately • The PC revolution! • Its amazing!

  15. TPC C improved fast(250%/year!) 40% hardware, 100% software, 100% PC Technology

  16. Problems • HSM (hierarchical storage management) • Design and Meta-data • Ingest • Data discovery, search, and analysis • reorganize-reprocess • disaster recovery • management/operations cost

  17. Demo http://msrlab/terraserver

  18. Outline • The challenge: Building GIANT data stores • for example, the EOS/DIS 15 PB system • Conclusion 1 • Think about Maps and SCANS • Conclusion 2: • Think about Clusters

  19. Meta-Message: Technology Ratios Are Important • If everything gets faster & cheaper at the same rate THEN nothing really changes. • Things getting MUCH BETTER: • communication speed & cost 1,000x • processor speed & cost 100x • storage size & cost 100x • Things staying about the same • speed of light (more or less constant) • people (10x more expensive) • storage speed (only 10x better)

  20. Today’s Storage Hierarchy : Speed & Capacity vs Cost Tradeoffs 15 4 10 10 12 2 10 10 9 0 10 10 6 -2 10 10 3 -4 10 10 Size vs Speed Price vs Speed Cache Nearline Tape Offline Main Tape Disc Secondary Online Online $/MB Secondary Tape Tape Typical System (bytes) Disc Main Offline Nearline Tape Tape Cache -9 -6 -3 0 3 -9 -6 -3 0 3 10 10 10 10 10 10 10 10 10 10 Access Time (seconds) Access Time (seconds)

  21. Storage Ratios Changed • 10x better access time • 10x more bandwidth • 4,000x lower media price • DRAM/DISK 100:1 to 10:10 to 50:1

  22. What's a Terabyte 1 Terabyte 1,000,000,000 business letters 100,000,000 book pages 50,000,000 FAX images 10,000,000 TV pictures (mpeg) 4,000 LandSat images Library of Congress (in ASCI) is 25 TB 1980: 200 M$ of disc 10,000 discs 5 M$ of tape silo 10,000 tapes 1997: 200 K$ of magnetic disc 120 discs 250 K$ of optical disc robot 200 platters 25 K$ of tape silo 25 tapes Terror Byte !! .1% of a PetaByte!!!!!!!!!!!!!!!!!! 150 miles of bookshelf 15 miles of bookshelf 7 miles of bookshelf 10 days of video

  23. The Cost of Storage & Access • File Cabinet: cabinet (4 drawer) 250$ paper (24,000 sheets) 250$ space (2x3 @ 10$/ft2) 180$ total 700$ 3 ¢/sheet • Disk: disk (9 GB =) 2,000$ ASCII: 5 m pages 0.2 ¢/sheet (15x cheaper • Image: 200 k pages 1 ¢/sheet (similar to paper)

  24. Trends: Application Storage Demand Grew • The New World: • Billions of objects • Big objects (1MB) • The Old World: • Millions of objects • 100-byte objects

  25. Trends:New Applications Multimedia: Text, voice, image, video, ... The paperless office Library of congress online (on your campus) All information comes electronically entertainment publishing business Information Network, Knowledge Navigator, Information at Your Fingertips

  26. Thesis: Performance =Storage Accesses not Instructions Executed • In the “old days” we counted instructions and IO’s • Now we count memory references • Processors wait most of the time

  27. The Pico Processor 1 M SPECmarks 106 clocks/ fault to bulk ram Event-horizon on chip. VM reincarnated Multi-program cache Terror Bytes!

  28. Storage Latency: How Far Away is the Data? Andromeda 9 10 Tape /Optical 2,000 Years Robot 6 Pluto Disk 2 Years 10 1.5 hr Sacramento 100 Memory This Campus 10 10 min On Board Cache 2 On Chip Cache This Room 1 Registers My Head 1 min

  29. The Five Minute Rule • Trade DRAM for Disk Accesses • Cost of an access (DriveCost / Access_per_second) • Cost of a DRAM page ( $/MB / pages_per_MB) • Break even has two terms: • Technology term and an Economic term • Grew page size to compensate for changing ratios. • Still at 5 minute for random, 1 minute sequential

  30. Shows Best Page Index Page Size ~16KB

  31. Standard Storage Metrics • Capacity: • RAM: MB and $/MB: today at 10MB & 100$/MB • Disk: GB and $/GB: today at 10 GB and 200$/GB • Tape: TB and $/TB: today at .1TB and 25k$/TB (nearline) • Access time (latency) • RAM: 100 ns • Disk: 10 ms • Tape: 30 second pick, 30 second position • Transfer rate • RAM: 1 GB/s • Disk: 5 MB/s - - - Arrays can go to 1GB/s • Tape: 5 MB/s - - - striping is problematic

  32. New Storage Metrics: Kaps, Maps, SCAN? • Kaps: How many kilobyte objects served per second • The file server, transaction processing metric • This is the OLD metric. • Maps: How many megabyte objects served per second • The Multi-Media metric • SCAN: How long to scan all the data • the data mining and utility metric • And • Kaps/$, Maps/$, TBscan/$

  33. For the Record (good 1997 devices) X 14

  34. How To Get Lots of Maps, SCANs • parallelism: use many little devices in parallel • Beware of the media myth • Beware of the access time myth At 10 MB/s: 1.2 days to scan 1,000 x parallel: 100 seconds SCAN. Parallelism: divide a big problem into many smaller ones to be solved in parallel.

  35. The Disk Farm On a Card The 100GB disc card An array of discs Can be used as 100 discs 1 striped disc 10 Fault Tolerant discs ....etc LOTS of accesses/second bandwidth 14" • Life is cheap, its the accessories that cost ya. • Processors are cheap, it’s the peripherals that cost ya • (a 10k$ disc card).

  36. Tape Farms for Tertiary StorageNot Mainframe Silos 100 robots 1M$ 50TB 50$/GB 3K Maps 10K$ robot 14 tapes 27 hr Scan 500 GB 5 MB/s 20$/GB Scan in 27 hours. many independent tape robots (like a disc farm) 30 Maps

  37. The Metrics: Disk and Tape Farms Win Data Motel: Data checks in, but it never checks out GB/K$ 1 , 000 , 000 Kaps 100 , 000 Maps SCANS/Day 10 , 000 1 , 000 100 10 1 0.1 0.01 1000 x D i sc Farm 100x DLT Tape Farm STC Tape Robot 6,000 tapes, 8 readers

  38. Tape & Optical: Beware of the Media Myth Optical is cheap: 200 $/platter 2 GB/platter => 100$/GB(2x cheaper than disc) Tape is cheap: 30 $/tape 20 GB/tape => 1.5 $/GB (100x cheaper than disc).

  39. Tape & Optical Reality: Media is 10% of System Cost • Tape needs a robot (10 k$ ... 3 m$ ) • 10 ... 1000 tapes (at 20GB each) => 20$/GB ... 200$/GB • (1x…10x cheaper than disc) • Optical needs a robot (100 k$ ) • 100 platters = 200GB ( TODAY ) => 400 $/GB • ( more expensive than mag disc ) • Robots have poor access times • Not good for Library of Congress (25TB) • Data motel: data checks in but it never checks out!

  40. The Access Time Myth The Myth: seek or pick time dominates The reality: (1) Queuing dominates (2) Transfer dominates BLOBs (3) Disk seeks often short Implication: many cheap servers better than one fast expensive server • shorter queues • parallel transfer • lower cost/access and cost/byte This is now obvious for disk arrays This will be obvious for tape arrays

  41. Outline • The challenge: Building GIANT data stores • for example, the EOS/DIS 15 PB system • Conclusion 1 • Think about Maps and SCAN & 5 minute rule • Conclusion 2: • Think about Clusters

  42. Scaleable ComputersBOTH SMP and Cluster Grow Up with SMP 4xP6 is now standard Grow Out with Cluster Cluster has inexpensive parts SMP Super Server Departmental Cluster of PCs Server Personal System

  43. What do TPC results say? • Mainframes do not compete on performance or price They have great legacy code (MVS) • PC nodes performance is 1/3 of high-end UNIX nodes • 6xP6 vs 48xUltraSparc • PC Technology is 3x cheaper than high-end UNIX • Peak performance is a cluster • Tandem 100 node cluster • DEC Alpha 4x8 cluster • Commodity solutions WILL come to this market

  44. Cluster Advantages • Clients and Servers made from the same stuff. • Inexpensive: Built with commodity components • Fault tolerance: • Spare modules mask failures • Modular growth • grow by adding small modules • Parallel data search • use multiple processors and disks

  45. Clusters being built • Teradata 500 nodes (50k$/slice) • Tandem,VMScluster 150 nodes (100k$/slice) • Intel, 9,000 nodes @ 55M$ ( 6k$/slice) • Teradata, Tandem, DEC moving to NT+low slice price • IBM: 512 nodes ASCI @ 100m$ (200k$/slice) • PC clusters (bare handed) at dozens of nodes web servers (msn, PointCast,…), DB servers • KEY TECHNOLOGY HERE IS THE APPS. • Apps distribute data • Apps distribute execution

  46. Clusters are winning the high end • Until recently a 4x8 cluster has best TPC-C performance • Clusters have best data mining story (TPC-D) • This year, a 32xUltraSparc cluster won the MinuteSort

  47. Clusters (Plumbing) • Single system image • naming • protection/security • management/load balance • Fault Tolerance • Hot Pluggable hardware & Software

  48. So, What’s New? New MPP & NewOS New MPP & NewOS New MPP & NewOS New MPP & NewOS New App New App New App New App • When slices cost 50k$, you buy 10 or 20. • When slices cost 5k$ you buy 100 or 200. • Manageability, programmability, usability become key issues (total cost of ownership). • PCs are MUCH easier to use and program MPP Vicious Cycle No Customers! Apps CP/Commodity Virtuous Cycle: Standards allow progress and investment protection Standard OS & Hardware Customers

  49. Where We Are Today • Clusters moving fast • OLTP • Sort • WolfPack • Technology ahead of schedule • cpus, disks, tapes,wires,.. • Databases are evolving • Parallel DBMSs are evolving • Operations (batch) has a long way to go on Unix/PC.

  50. Outline • The challenge: Building GIANT data stores • for example, the EOS/DIS 15 PB system • Conclusion 1 • Think about Maps and SCANs & 5 minute rule • Conclusion 2: • Think about Clusters • Slides & paper: http:\\www.research.Microsoft.com\~Gray\talks December SIGMOD RECORDhttp:\\www.research.Microsoft.com\~Gray\5_Min_Rule_Sigmod.doc

More Related