1 / 15

Interconnection Architectures for Petabyte-Scale Storage Systems

This literature review explores high-performance storage systems designed for petabyte-scale data storage, emphasizing the need for scalable performance and fault tolerance. It delves into interconnection architectures, including network topologies like Fat Tree, Butterfly, Mesh, Torus, and Hypercube, and analyzes their bandwidth capabilities. The study proposes building commodity-based storage fabrics using a mix of lower-speed and high-speed links to enhance reliability and performance, with a focus on using higher-dimension torus topologies for achieving 100GB/s throughput from 1PB systems.

cynara
Download Presentation

Interconnection Architectures for Petabyte-Scale Storage Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Literature Review Interconnection Architectures for Petabye-Scale High-Performance Storage Systems Andy D. Hospodor, Ethan L. Miller IEEE/NASA Goddard Conference on Mass Storage Systems and Technologies April 2004 Henry Chen September 24, 2010

  2. Introduction • High-performance storage systems • Petabytes (250 bytes) of data storage • Supply hundreds or thousands of compute nodes • Aggregate system bandwidth >100GB/s • Performance should scale with capacity • Large individual storage systems • Require high-speed network interface • Concentration reduces fault tolerance

  3. Proposal • Follow high-performance computing evolution • Multi-processor networks • Network of commodity devices • Use disk + 412port 1GbE switch as building block • Explore & simulate interconnect topologies

  4. Commodity Hardware • Network • 1Gb Ethernet: ~$20 per port • 10Gb Ethernet: ~$5000 per port (25x per Gb per port) • Aside: Now ~$1000 per port • Disk drive • ATA/(SATA) • FibreChannel/SCSI/(SAS)

  5. Setup • Target 100GB/s bandwidth • Build system using 250GB drives (2004) • 4096 drives to reach 1PB • Assume each drive has 25MB/s throughput • 1Gb link supports 23 disks • 10Gb link supports ~25 disks

  6. Basic Interconnection • 32 disks/switch • Replicate system 128x • 4096 1Gb ports • 128 10Gb ports • ~Networked RAID0 • Data local to each server

  7. Fat Tree • 4096 1Gb ports • 2418 10Gb ports • 2048 switch to router(128 Sw × 8 Rt × 2) • 112 inter-router • 256 server to router (×2) • Need large, multi-stage routers • ~$10M for 10Gb ports

  8. Butterfly Network • Need “concentrator” switch layer • Each network level carries entire traffic load • Only one path between any two server and storage

  9. Mesh • Routers to servers atmesh edges • 16384 1Gb links • Routers only atedges; mesh providespath redundancy

  10. Torus • Mesh with edgeswrapped around • Reduces average pathlength • No edges; dedicatedconnection breakoutto servers

  11. Hypercube • Special-case torus • Bandwidth scalesbetter than mesh/torus • Connections per nodeincreases with system • Can group devices intosmaller units and connectwith torus

  12. Bandwidth • Not all topologies actually capable of 100GB/s • Maximum simultaneous bandwidth Link speed × number of links Average hops

  13. Analysis • Embedding switches in storage fabric uses fewer high-speed ports, but more low-speed ports

  14. Router Placement in Cube-Styles • Routers require nearly 100% bandwidth of links • Adjacent routers cause overload & underload • Use random placement; optimization possible?

  15. Conclusions • Build multiprocessor-style network for storage • Commodity-based storage fabrics can be used to improve reliability and performance; scalable • Rely on large number of lower-speed links; limited number of high-speed links where necessary • Higher-dimension torii (4-D, 5-D) provides reasonable solution for 100GB/s from 1PB

More Related