how computer architecture trends may affect future distributed systems
Download
Skip this Video
Download Presentation
How Computer Architecture Trends May Affect Future Distributed Systems

Loading in 2 Seconds...

play fullscreen
1 / 41

How Computer Architecture Trends May Affect Future Distributed ... - PowerPoint PPT Presentation


  • 409 Views
  • Uploaded on

How Computer Architecture Trends May Affect Future Distributed Systems. Mark D. Hill Computer Sciences Department University of Wisconsin--Madison http://www.cs.wisc.edu/~markhill PODC ‘00 Invited Talk. Three Questions. What is a System Area Network (SAN) and how will it affect clusters?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'How Computer Architecture Trends May Affect Future Distributed ...' - betty_james


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
how computer architecture trends may affect future distributed systems

How Computer Architecture TrendsMay Affect Future Distributed Systems

Mark D. Hill

Computer Sciences Department

University of Wisconsin--Madison

http://www.cs.wisc.edu/~markhill

PODC ‘00 Invited Talk

three questions
Three Questions
  • What is a System Area Network (SAN)and how will it affect clusters?
    • E.g., InfiniBand
  • How fat will multiprocessor servers beand how to we build larger ones?
    • E.g. Wisconsin Multifacet’s Multicast & Timestamp Snooping
  • Future of multiprocessor servers & clusters?
    • A merging of both?
outline
Outline
  • Motivation
  • System Area Networks
  • Designing Multiprocessor Servers
  • Server & Cluster Trends
technology push moore s law
Technology Push: Moore’s Law
  • What do following intervals have in common?
    • Prehistory to 2000
    • 2001 to 2002
  • Answer: Equal progress in absolute processor speed(and more doubling 2003-4, 2005-6, etc.)
    • Consider salary doubling
  • Corollary: Cost halves every two years
    • Jim Gray: In a decade you can buy a computerfor less than its sales tax today
application pull
Application Pull
  • Should use computers in currently wasteful ways
    • Already computers in electric razors & greeting cards
  • New business models
    • B2C, B2B, C2B, C2C
    • Mass customization
  • More proactive (beyond interactive) [Tennenhouse]
    • Today: P2C where P==Person & C==Computer
    • More C2P: mattress adjusts to save your back
    • More C2C: Agents surf the web for optimal deal
    • More sensors (physical/logic worlds coupled)
    • More hidden computers (c.f., electric motors)
  • Furthermore, I am wrong
the internet iceberg
The Internet Iceberg
  • Internet Components
    • Clients -- mobile, wireless
    • “On Ramp” -- LANs/DSL/Cable Modems
    • WAN Backbone -- IPv6, massive BW
    • and ...
  • SERVICES
    • Scale Storage
    • Scale Bandwidth
    • Scale Computation
    • High Availability
outline7
Outline
  • Motivation
  • System Area Networks
    • What is a SAN?
    • InfiniBand
    • Virtualizing I/O with Queue Pairs
    • Predictions
  • Designing Multiprocessor Servers
  • Server & Cluster Trends
regarding storage bandwidth

proc

proc

memory interconnect

memory

Regarding Storage/Bandwidth
  • Currently resides on I/O Bus (PCI)
    • HW & SW protocol stacks
    • Must add hosts to add storage/bandwidth

bridge

i/o bus

i/o slot 0

i/o slot n-1

want system area network san
Want System Area Network (SAN)
  • SAN vs. Local Area Nework (LAN)
    • Higher bandwidth (10 Gbps)
    • Lower latency (few microseconds or less)
    • More limited size
    • Other (e.g., single administrative domain, short distance)
    • Examples: Tandem Servernet & Myricom Myrinet
  • Emerging Standard: InfiniBand
    • www.inifinibandTA.org w/ spec 1.0 Summer 2000
    • Compaq, Dell, HP, IBM, Intel, Microsoft, Sun, & others
    • 2.5 Gbits/s times 1, 4, or 12 wires
infiniband model from website

proc

proc

memory interconnect

(host channel adapter)

memory

HCA

link

Othernetworks

router

XCA

switch

Other switches, hosts, targets, etc.

InfiniBand Model (from website)

target

(disks)

TCA

inifiniband advantages
Inifiniband Advantages
  • Storage/Network made orthogonal from Computation
  • Reduce “hardware” stack -- no i/o bridge
  • Reduce “software” stack; hardware support for
    • Connected Reliable
    • Connected Unreliable
    • Datagram
    • Reliable Datagram
    • Raw Datagram
  • Can eliminate system call for SAN use (next slide)
virtualizing infiniband
Virtualizing InfiniBand
  • I/O traditionally virtualized with system call
    • System enforces isolation
    • System permits authorized sharing
  • Memory virtualized
    • System trap/call for setup
    • Virtual memory hardware for common-case translation
  • Infiniband exploits “queue pairs” (QPs) in memory
    • C.f., Intel Virtual Interface Architecture (VIA)[IEEE Micro, Mar/Apr ‘98]
    • Users issue sends, receives, & remote DMA reads/writes
queue pair
Queue Pair
  • QP setup system call
    • Connect with process
    • Connect with remote QP(not shown here)
  • QP placed in “pinned” virtual memory
  • User directly access QP
    • E.g., sends, receives & remote DMA reads/writes

proc

Main

Memory

dma-W4

dma-R3

send2

receive1

send1

receive2

HCA

infiniband cont
InfiniBand, cont.
  • Roadmap
    • NGIO/FIO merger in ‘99
    • Spec in ‘00
    • Products in ‘03-’10
  • My Assessment
    • PCI needs successor
    • InfiniBand has the necessary features (but also many others)
    • InifiniBand has considerable industry buy-in (but it is recent)
    • Gigabit Ethernet will be only competitor
      • Good name with backing from Cisco et al.
      • But TCP/IP is a killer
    • Infiniband for storage will be key
infiniband research issues
InfiniBand Research Issues
  • Software Wide Open
    • Industry will do local optimization(e.g., still have device driver virtualized with system calls)
    • But what is the “right” way to do software?
    • Is there a theoretical model for this software?
  • Other SAN Issues
    • A theoretical model of a service-providers site?
    • How to trade performance and availability?
    • Utility of broadcast or multicast support?
    • Obtaining quasi-real-time performance?
outline16
Outline
  • Motivation
  • System Area Networks
  • Designing Multiprocessor Servers
    • How Fat?
    • Coherence for Servers
    • E.g., Multicast Snooping
    • E.g., Timestamp Snooping
  • Server & Cluster Trends
how fat should servers be

Servers running

databases for

“hard” state

PCs w/

“soft” state

How Fat Should Servers Be?
  • Use
    • PCs -- cheap but small
    • Workgroup servers -- medium cost; medium size
    • Large servers -- premium cost & size
  • One answer: “yes”
how do we build the big servers
How Do We Build the Big Servers?
  • (Industry knows how to build the small ones)
  • A key problem is the memory system
    • Memory Wall: E.g., 100ns memory access =400 instruction opportunities for 4-way 1GHz processor
  • Use per-processor caches to reduce
    • Effective Latency
    • Effective Bandwidth Used
  • But cache coherence problem ...
coherence 101

Pn-1

P0

P1

“?”

“?”

r2<-m[100]

r3<-m[100]

cache

cache

cache

Coherence 101

“4”

“4”

r0<-m[100]

r1<-m[100]

m[100]<-5

X 5

100 : 4

100 : 4

interconnection network

memory

memory

100 4

broadcast snooping

P2:GETX

Ordered Address Network

P1:GETX

P1:GETX

P2:GETX

Mem

P0

P1

P2

P1:GETX

P2:GETX

P1:GETX

P2:GETX

P1:GETX

P2:GETX

P1:GETX

P2:GETX

data

data

Data Network

Broadcast Snooping

P2:GETX

P2:GETX

data

data

data

broadcast snooping21
Broadcast Snooping
  • Symmetric Multiprocessor (SMP)
    • Most commercially-successful parallel computer architecture
    • Performs well by finding data directly
    • Scales poorly
  • Improvements, e.g., Sun E10000
    • Split address & data transactions
    • Split address & data network (e.g., bus & crossbar)
    • Multiple address buses (e.g., four multiplexed by address)
    • Address bus is broadcast tree (not shared wires)
  • But…
    • Broadcast all address transactions (expensive)
    • All processors must snoop all transactions
directories

Address Network

P1:GETX

P2:GETX

send

send

Dir/Mem

P0

P1

P2

data

data

Data Network

Directories

P2:GETX

P1:GETX

P2:GETX

data

data

data

directories23
Directories
  • Directory Based Cache Coherence
    • E.g., SGI/Cray Origin2000
    • Allows arbitrary point-to-point interconnection network
    • Scales up well
  • But
    • Cache-to-cache transfers common in demanding apps(55-62% sharing misses for OLTP [Barroso ISCA ‘98])
    • Many applications can’t use 100s of processors
    • Must also “scale down” well
wisconsin multifacet big picture
Wisconsin Multifacet: Big Picture
  • Build Servers For Internet economy
    • Moderate multiprocessor sizes: 2-8 then 16-64, but not 1K
    • Optimize for these workloads (e.g. cache-to-cache transfers)
  • Key Tool: Multiprocessor Prediction & Speculation
    • Make a guess... verify it later
    • Uniprocessor predecessors: branch & set predictors
    • Recent multiprocessor work: [Mukherjee/Hill ISCA98], [Kaxiras/Goodman HPCA99] & [Lai/Falsafi ISCA99]
    • Multicast Snooping
    • Timestamp Snooping
comparison of coherence methods
Comparison of Coherence Methods

Use prediction to improve on both?

multicast snooping
Multicast Snooping
  • On cache miss
    • Predict "multicast mask" (e.g., bit vector of processors)
    • Issue transaction on multicast address network
  • Networks
    • Address network that totally-orders address multicasts
    • Separate point-to-point data network
  • Processors snoop all incoming transactions
    • If it\'s your own, it "occurs" now
    • If another\'s, then invalidate and/or respond
  • Simplified directory (at memory)
    • Purpose: Allows masks to be wrong (explained later)
predicting masks
Predicting Masks
  • Performed at Requesting Processor
    • Include owner (GETS/GETX) & all sharers (GETX only)
    • Exclude most other processors
  • Techniques
    • Many straightforward cases (e.g., stack, code,space-sharing)
    • Many options (network load, PC, software, local/global)

predicted mask

block address

Mask Predictor

feedback

implementing an ordered multicast network
Implementing an Ordered Multicast Network
  • Address Network
    • Must create the illusion of total order of multicasts
    • May deliver a multicast to destinations at different times
  • Wish List
    • High throughput for multicasts
    • No centralized bottlenecks
    • Low latency and cost (~ pipelined broadcast tree)
    • ...
  • Sample Solutions
    • Isotach Networks [Reynolds et al., IEEE TPDS 4/97]
    • Indirect Fat Tree [ISCA `99]
    • Direct Torus
indirect fat tree cont
Indirect Fat Tree, cont.
  • Basic Idea
    • Processors send transactions up to roots
    • Roots send transactions down with logical timestamp
    • Switches stall transactions to keep in order
    • Null transaction sent to avoid deadlock
  • Assessment
    • Viable & high cross-section bandwidth
    • Many "backplane" ASICs means higher cost
    • Often stalls transactions
  • Want
    • Lower cost of direct connections
    • Always delivery transactions as soon as possible (ASAP)
    • Sacrifice some cross-section bandwidth
direct 2 d torus work in progress
Direct 2-D Torus (work in progress)
  • Features
    • Each processor is switch
    • Switches directly connected
    • E.g., network of Compaq 21364
  • Network order?
    • Broadcasts unordered
    • Snooping needs total order
  • Solution
    • Create order with logical timestampsinstead of network delivery order
    • Called Timestamp Snooping [ASPLOS ‘00]

0

1

14

15

timestamp snooping
Timestamp Snooping
  • Timestamp Snooping
    • Snooping with order determined by logical timestamps
    • Broadcast (not multicast) in ASPLOS ‘00
  • Basic Idea
    • Assign timestamp to coherence transactions at sender
    • Broadcast transactions over unordered network ASAP
    • Transaction carry timestamp (2 bits)
    • Processors process transactions in timestamp order
timestamp snooping issues
Timestamp Snooping Issues
  • More address bandwidth
    • For 16-processors, 4-ary butterfly, 64-byte blocks
    • Directory: 3*8 + 3*72 + more = 240 + more
    • Timestamp Snooping 21*8 + 3*72 = 384 (< 60% more)
  • Network must guarantee timestamps
    • Assert future transactions will have greater timestamps(so processor can processor older transactions)
    • Isotach [Reynolds IEEE TPDS 4/97] more aggressively
  • Other
    • Priority queue at processor to order transactions
    • Flow control and buffering issues
initial multifacet results
Initial Multifacet Results
  • Multicast Snooping [ISCA ‘99]
    • Ordered multicast of coherence transactions
    • Find data directly from memory or caches
    • Reduce bandwidth to permit some scaling
    • 32-processor results show 2-6 destinations per multicast
  • Timestamp Snooping [ASPLOS ‘00]
    • Broadcast snooping with “order” determined by logical timestamps carried by coherence transactions
    • No bus: Allows arbitrary memory interconnects
    • No directory or directory indirection
    • 16-processor results show 25% faster for 25% more traffic
selected issues
Selected Issues
  • Multicast Snooping
    • What program property are mask predictors exploiting?
    • Why is there no good model of localityor the “90-10” rule in general?
    • How does one build multicast networks?
    • What about fault tolerance?
  • Timestamp Snooping
    • What is an optimal network topology?
    • What about buffering, deadlock, etc.?
    • Implementing switches and priority queues?
outline36
Outline
  • Motivation
  • System Area Networks
  • Designing Multiprocessor Servers
  • Server & Cluster Trends
    • Out-of-box and highly-available servers
    • High-performance communication for clusters
multiprocessor servers

SMP

SMP

SMP

SMP

Multiprocessor Servers
  • High-Performance Communication “within box”
    • SMPs (e.g., Intel PentiumPro Quads)
    • Directory-based (SGI Origin2000)
  • Trend toward hierarchical “out of box” solutions
    • Build bigger servers from smaller ones
    • Intel Profusion, Sequent NUMA-Q, Sun WildFire (pictured)
multiprocessor servers cont
Multiprocessor Servers, cont.
  • Traditionally had poor error isolation
    • Double-bit ECC error crashes everything
    • Kernel error crashes everything
    • Poor match for highly available Internet infrastructure
  • Improve error isolation
    • IBM 370 “virtual machines”
    • Stanford HIVE “cells”
clusters
Clusters
  • Traditionally
    • Good error isolation
    • Poor communication performance (especially latency)
    • LANs are not optimized for clusters
  • Enter Early SANs
    • Berkeley NOW w/ Myricom Myrinet
    • IBM SP w/ proprietary network
  • What now with InfiniBand SAN (or alternatives)?
a prediction
A Prediction
  • Blurring of cluster & server boundaries
  • Clusters
    • High communication performance
  • Servers
    • Better error isolation
    • Multi-box solutions
  • Use same hardware & configure in the field
  • Issues
    • How do we model these hybrids?
    • Should PODC & SPAA also converge?
three questions41
Three Questions
  • What is a System Area Network (SAN)and how will it affect clusters?
    • E.g., InfiniBand
    • Make computation, storage, & network orthogonal
  • How fat will multiprocessor servers beand how to we build larger ones?
    • Varying sizes for soft & hard state
    • E.g., Multicast Snooping & Timestamp Snooping
  • Future of multiprocessor servers & clusters?
    • Servers will support higher availability & extra-box solutions
    • Clusters will get server communication performance
ad