A Physical Perspective of Computer Architecture

1. A Physical Perspective of Computer Architecture Peter Hsu, Ph.D. Chief Architect Microprocessor Development Toshiba America Electronics Components, Inc. Presented February 13, 2001 at Univ. of Wisconsin in Madison

2. Introduction • Computer Architecture Perspectives • Logical, Performance • Abstract • Quantitative • Academia • Physical, Cost • Constrained by History, Emotions, Physics • Modulated by Current World Affairs • Apprenticeship A Physical View of Computer Architecture

3. Content • Tour a Particular Design Point • Rationale of Choices • Confluence of Decisions • Rules of Thumb Computer Markets • PC Infrastructure Defines Most Cost-Efficient • Consumer Volume Advancing Technology • All other Computers Competing with PC A Physical View of Computer Architecture

4. heat distributor 88 chip stacks 10mm alignment cage silicon substrate 14cm 12mm 4mm printed circuit board  3000 wire bonds pressure plate Multichip Module A Physical View of Computer Architecture

5. 10m width 20m pitch 12mm Chip Stack 12mm 0.3mm router 10mm DRAMs processors stack shown upside down A Physical View of Computer Architecture

6. Stack to Substrate Connection wirebond springs heat conventional wirebond pads DRAMs router chip silicon substrate A Physical View of Computer Architecture

7. System Unit heat sink multichip module heat pipe flex signal PCB 1.5 in input/output connectors rigid power PCB 19 inches A Physical View of Computer Architecture

8. office (110V 15A) Scalable Configurations peripherals system units peripherals 64-node supercomputer (80 Kilowatt) copier room (220V 30A) A Physical View of Computer Architecture

9. CPU CPU CPU DRAMs DRAMs DRAMs router router router chip stack chip stack chip stack silicon multichip substrate system unit Physical Architecture cables between system units A Physical View of Computer Architecture

10. CPU CPU CPU CPU L1$ L1$ L1$ L1$ Logical Architecture chip stack level 2 cache main memory router byte-wide point-to-point network system unit serial point-to-point cable network A Physical View of Computer Architecture

11. Guiding Principles • Performance 1.  Latency (Memory, Interprocessor, etc.), 2.  Bandwidth, then 3.  Microarchitecture • Cost • Silicon Portion Scales With Process (e.g. Learning curve of copper-on-silicon substrate) • Non-Silicon Portion Does Not Scale (e.g. Liquid immersion cooling hardware) A Physical View of Computer Architecture

12. Silicon Substrate 4mm 12mm maximum trace length 24.8cm 12mm chip maximum cutset 2048 p-to-p links 4mm spacer 200mm (8 inch) wafer 150m pitch 3200 wire bonds substrate to PCB 14cm A Physical View of Computer Architecture

13. 75m tolerance Stack to Substrate Connection alignment cage substrate chip stack 250m pitch 75m clearance (0.003 inch or 3 mils) 125m pad 125m space Rule of thumb chip stack • Machined parts need several mils tolerance A Physical View of Computer Architecture

14. Substrate Design • Internal Signals • Link • 8 data, 2 clock bits (20% overhead) • Source Synchronous • Density • 20,480 signals across cutset ( 7m per track) • 63210  1260 signals / stack (2304 total) Rule of thumb • High speed  50% signal pads A Physical View of Computer Architecture

15. Substrate Design (con’t) • External Connections • Signal • Node to multicomputer node(2in 2out 64) • Node to peripheral device (2in 2out 64) • Power • 1280 power/ground pairs • 20W per stack (VDD  1V) Rule of thumb • A wire bond  1A sustained current A Physical View of Computer Architecture

16. Interconnect Dimensions width W space S pitch 4 3.5 7.5 height H 6 VDD insulation thickness T 8 VSS 2 3 4.5 5 10 5.5 15 7.5 A Physical View of Computer Architecture

17. ( ) ( ) 0.222 0.222 ( ) ( ) ( ) 0.222 W T H T C  W T H T H T = 1.15 + 2.80 [ ] + 0.06 + 1.66  0.14 L W  H   C0 C ( ) 1.34 R =  Z0 = T S Electrical Characteristics Bakoglu, H.B., Circuits, Interconnections, and Packaging for VLSI,Addison-Wesley, 1990. A Physical View of Computer Architecture

18. 1 1 1 V V V 0 0 0 time time time Lossy Transmission Line Self terminating if Z0 R  2Z0 A Physical View of Computer Architecture

19. Substrate Design (con’t) • Construction • Material • Copper,   1.7 mcm • “Low-k” Insulator,   3.0 • Design Rules • 1 L 7.2cmR 51Z0 27 • 2L 18.4cmR 52Z0 26 • 3L 24.8cmR 47Z0 27 • 7 Layers (3 X•Y  pad) A Physical View of Computer Architecture

20. Package Design • Thermal • System Unit • Ambient: A  40C • Airflow: 1 m/s (200 ft/min) • 1280W  16  16  1in Rule of thumb • Heat sink with fan dissipates 5W per inch3 A Physical View of Computer Architecture

21. gas condense boil liquid metal wick heat pipe heat Package Design (con’t) • Thermal Resistance • DRAM leakage: J  80C • 1280W,   40C JA  0.03C/W Rule of thumb • Solid heat sink JA  1C/W A Physical View of Computer Architecture

22. router 3W logic + 1W substrate (4W total) 41W active simultaneously (4W total) DRAMs 42.5W CPU + 2W L2 cache (12W total) processors Thermal Limits • Major Design Implications • PC Processor 10-30W • First Order Constraint • MHz • Latencies Observation • Activity vs. State Retention Density A Physical View of Computer Architecture

23. Package Design (con’t) 10% variation • Power Supply • PC  10¢ / W • Server  30¢ / W • Exotic  $1 / W 15A 110V AC -5% first stage 120A 12V DC -10% 1,280A 1V DC second stage Rule of thumb • Standard 15A, 110V AC outlet  1300W A Physical View of Computer Architecture

24. 1 inch Package Design (con’t) • Cable Connectors • Serial, e.g. USB • 64 per side • Finger Access, Airflow flex signal PCB Rule of thumb • Connector 0.3 in2 panel, 0.7 in2 clearance A Physical View of Computer Architecture

25. FSB NB addr data NB FSB R R R S R R S R R S R R S R global global global global cell array cell array cell array cell array DQ DQ DQ DQ T B 3m cable B T R S R R S R T B 3m cable B T Memory Latencies 7.5ns 37.5ns 7 PC133 “3-2-3” 82.5ns 22ns 29 Stack 58ns substrate Substrate 76ns PCB trace transceiver Cable 176ns Rule of thumb • PCB  10cm/ns (5ns/foot), coax 20cm/ns A Physical View of Computer Architecture

26. Memory Latencies (con’t) • Benefits • Performance • Cache miss penalty • Robustness • Global vs. local memory  30% • Remote access  3 local • Marketability • Minimize application speed variance • “No surprises” A Physical View of Computer Architecture

27. High Lights • Scalability • Partial node ... 64-node supercomputer • Performance • Latency • PC133-timing to 1 TBytes • Bandwidth • 32B / stack / cycle on substrate • 1/6B / stack / cycle via cable A Physical View of Computer Architecture

28. High Lights (con’t) • Risk Management • Not liquid immersion; no pumps, hoses • Configurable for 110V outlet • Substrate uses ordinary silicon process Taken Risks • Stacking chips not mainstream • Wirebond spring recent invention • Heat pipe reliability A Physical View of Computer Architecture

29. Summary • Architecture of a Large Computer • Performance • Materials • Mechanical Assembly • Thermal Management • Power Supply • Many More Issues... • Architect responsible for everything, even if s/he doesn’t know anything about it! A Physical View of Computer Architecture