Networks of Workstations

1. Networks of Workstations Electrical Engineering 864 Advanced Computer Design Prabal Dutta dutta.4@osu.edu Networks of Workstations

2. Introduction • Much of this class has been about the details of advanced and parallel computing. • Interconnection Networks • Routing. • Pipelining. • Scheduling. • Algorithms. • The purpose of this presentation is to look at the trends in parallel computing with a focus on networking… • And to understand the technical and economic forces driving changes in parallel computing today. Networks of Workstations

3. Taxonomy • Massively Parallel Processors (MPP) • Built with specialized networks • Specifically intended for use as parallel computers • Clusters • Clusters: small number of processors/node. • Constellations: large number of processors/node. • Networks of Workstations (NOW) • Scalable interconnection networks. • Clusters of commodity PCs/workstations and high-performance network gear. • Beowulf = Linux + Bonded Ethernet or Myrinet. • UCB NOW: HP9000 + FDDI, ATM, or Myrinet. • Other • Single Instruction Multiple Data: Connection Machine 2 • Vector Processors: Cray • Let’s take a look at some of the once-high-flyers of MPPs… Networks of Workstations

4. Bankrupt, Acquired, Spun Out, Sold Off! Networks of Workstations

5. What Happened? • In the 1990s, the supercomputing market had completely changed: "The cold war was over, the commercial world was so competitive, and companies couldn’t afford the machines. Meanwhile, microprocessors became more powerful. It was the beginning of commodity technology instead of proprietary technology. There was the move to vertically integrate everything. We were in a new age. Supercomputers were too expensive. They were not only hard to build, but they took a long time to build.” - Steve Chen, Ph.D. 1975, UIUC Chief Architect of the Cray X-MP Networks of Workstations

6. Granularity: Computation-Communications Tradeoff $$$ $$ Distributed Computing MPP Constellations Workstation Computing Computations Networks of Workstations Clusters Desktop Computing SMP $$ $ Communications Networks of Workstations

7. Top 500 Supercomputer Sites (Metric: GFlops) Source data: http://www.top500.org/ Networks of Workstations

8. Top 500 Supercomputer Sites (Metric: Processor Arch) Source data: http://www.top500.org/ Networks of Workstations

9. Why are MPPs Falling Behind? • Massively Parallel Processors • Cost too much. • Required over-engineering due to high integration. • Focused on niche applications with small markets. • Did not take into account networking commoditization. • Always a year or two behind the current state of the art. • On the other hand, Networks of Workstations • Were marginally more expensive than workstations. • Leveraged emerging fast, scalable, and switched LANs • Leveraged high-volume, low-cost, super-pipelined commodity processors. • Extended existing operating systems with global process management and shared memory/message passing abstractions. Networks of Workstations

10. Emergence of the “Killer” Network • Massively Parallel Processing took advantage of: • Switched Networks. • Scalable bandwidth. • Low-latency communications. • Low processor overhead. • These technologies are becoming available in LANs: • Myrinet. • Used in 140 of the top 500 supercomputers. • Asynchronous Transfer Mode (ATM). • Fiber Distributed Data Interconnect (FDDI). • Fast Ethernet/Gigabit Ethernet. Networks of Workstations

11. Fast, Scalable, Switched Networks: Myrinet • Myrinet is based on the networks from MPPs and is used in 140 of the top 500 supercomputers. • By decoupling network from workstation, allows each to evolve at its own pace. • General Features • Full-duplex 2x2Gbps data rate (250MBps). • Self-initializing. • Low-latency. • Cut-through crossbar switches. • Direct communications between user processes and network. • Scales to tens of thousands of hosts. • Provide alternative communication paths between hosts. • X-Y Dimension-order routing. • Flow control on every link obviates packet buffering. • Error control and "heartbeat" continuity monitoring every link. • Linux,Windows,Solaris,Mac OSX,True64, FreeBSB,VxWorks Networks of Workstations

12. Fast, Scalable, Switched Networks: Myrinet • Sustained One-Way Data Rates Networks of Workstations

13. Fast, Scalable, Switched Networks: Myrinet • Example Myrinet Topologies Networks of Workstations

14. Fast, Scalable, Switched Networks: Myrinet • PCI Host Interface Card: $995 • Modular Switch Enclosure: $12,800 • 8-port Fiber Switch Line Card: $2,400 Affordable !!! Networks of Workstations

15. The “Killer” Workstation (and Desktop) • Workstations have become extraordinarily powerful and affordable. • Example: Intel Pentium 4 - 3.06GHz System: <$1,200 Networks of Workstations

16. The I/O Bottleneck • Processors are getting faster and disks are improving mostly in capacity, not performance. • So…we’ve reached a point of diminishing returns • More and more time spent waiting for I/O. • Little benefit for the end user. • Fast networks enable aggregate DRAM as a giant cache for disks. • Network DRAM access time is 1/10 of local disk. • I/O can be striped across multiple nodes like RAID. Networks of Workstations

17. Key Lessons • Never bet against the power of commoditization. • Market forces are unstoppable. • Frequently, we must decide between novel research or customer needs. • Companies formed to commercialize innovative research without compelling market economics are doomed • There are plenty of good research problems. • There are plenty of good business opportunities. • There are far fewer good research problems that are also good business opportunities. Networks of Workstations

18. Closing Thoughts Networks of Workstations

19. References • T. Anderson, D. Culler, and D. Patterson, “A Case of NOW (Networks of Workstations),” IEEE Micro, Vol. 15, Issue 1, February 1995. • T. Anderson, D. Culler, and D. Patterson, “The Berkeley Networks of Workstations (NOW) Project,” CompCon ’95, March 1995. • N. Boden, D. Cohen, R. Feldman, A. Kulawik, C. Seitz, J. Seizovic, and W. Su, “Myrinet: A Gigabit-per-Second Local Area Network,” IEEE Micro, Vol. 15, Issue 1, February 1995. • Douglas P. Ghormley, David Petrou, Steven H. Rodrigues, Amin M. Vahdat, and Thomas E. Anderson, “GLUnix: A Global Layer Unix for a Network of Workstations,” Software Practice and Experience, Special Issue on Experience with Distributed Systems, 1998. • http://www.myrinet.com • http://www.columbusmicro.com/ Networks of Workstations