1 / 75

CS252 Graduate Computer Architecture Lecture 23 I/O Continued

CS252 Graduate Computer Architecture Lecture 23 I/O Continued. November 17, 1999 Prof. John Kubiatowicz. Review: Disk Device Terminology. Disk Latency = Queuing Time + Seek Time + Rotation Time + Xfer Time + Ctrl Time. Order of magnitude times for 4K byte transfers:.

melaniemorgan
Download Presentation

CS252 Graduate Computer Architecture Lecture 23 I/O Continued

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS252Graduate Computer ArchitectureLecture 23I/O Continued November 17, 1999 Prof. John Kubiatowicz

  2. Review: Disk Device Terminology Disk Latency = Queuing Time + Seek Time + Rotation Time + Xfer Time + Ctrl Time Order of magnitude times for 4K byte transfers: Seek: 12 ms or less Rotate: 4.2 ms @ 7200 rpm = 0.5 rev/(7200 rpm/60m/s) (8.3 ms @ 3600 rpm ) Xfer: 1 ms @ 7200 rpm (2 ms @ 3600 rpm) Ctrl: 2 ms (big variation) Disk Latency = Queuing Time + (12 + 4.2 + 1 + 2)ms = QT + 19.2ms Average Service Time = 19.2 ms

  3. Queue Proc IOC Device But: What about queue time?Or: why nonlinear response Response Time (ms) 300 Metrics: Response Time Throughput 200 100 0 100% 0% Throughput (% total BW) Response time = Queue + Device Service time

  4. Departure to discuss queueing theory (On board)

  5. Introduction to Queueing Theory • More interested in long term, steady state than in startup => Arrivals = Departures • Little’s Law: Mean number tasks in system = arrival rate x mean reponse time • Observed by many, Little was first to prove • Applies to any system in equilibrium, as long as nothing in black box is creating or destroying tasks Arrivals Departures

  6. System server Queue Proc IOC Device A Little Queuing Theory: Notation • Queuing models assume state of equilibrium: input rate = output rate • Notation: r average number of arriving customers/secondTser average time to service a customer (tradtionally µ = 1/ Tser)u server utilization (0..1): u = r x Tser (or u= r / Tser )Tq average time/customer in queue Tsys average time/customer in system: Tsys = Tq + TserLq average length of queue: Lq = r x Tq Lsys average length of system: Lsys = r x Tsys • Little’s Law: Lengthsystem = rate x Timesystem (Mean number customers = arrival rate x mean service time)

  7. System server Queue Proc IOC Device A Little Queuing Theory • Service time completions vs. waiting time for a busy server: randomly arriving event joins a queue of arbitrary length when server is busy, otherwise serviced immediately • Unlimited length queues key simplification • A single server queue: combination of a servicing facility that accomodates 1 customer at a time (server) + waiting area (queue): together called a system • Server spends a variable amount of time with customers; how do you characterize variability? • Distribution of a random variable: histogram? curve?

  8. System server Queue Proc IOC Device A Little Queuing Theory • Server spends a variable amount of time with customers • Weighted mean m1 = (f1 x T1 + f2 x T2 +...+ fn x Tn)/F (F=f1 + f2...) • variance = (f1 x T12 + f2 x T22 +...+ fn x Tn2)/F – m12 • Must keep track of unit of measure (100 ms2 vs. 0.1 s2 ) • Squared coefficient of variance: C = variance/m12 • Unitless measure (100 ms2 vs. 0.1 s2) • Exponential distribution C = 1: most short relative to average, few others long; 90% < 2.3 x average, 63% < average • Hypoexponential distributionC < 1: most close to average, C=0.5 => 90% < 2.0 x average, only 57% < average • Hyperexponential distributionC > 1: further from average C=2.0 => 90% < 2.8 x average, 69% < average Avg.

  9. System server Queue Proc IOC Device A Little Queuing Theory: Variable Service Time • Server spends a variable amount of time with customers • Weighted mean m1 = (f1xT1 + f2xT2 +...+ fnXTn)/F (F=f1+f2+...) • Squared coefficient of variance C • Disk response times C ­ 1.5 (majority seeks < average) • Yet usually pick C = 1.0 for simplicity • Another useful value is average time must wait for server to complete task: m1(z) • Not just 1/2 x m1 because doesn’t capture variance • Can derive m1(z) = 1/2 x m1 x (1 + C) • No variance => C= 0 => m1(z) = 1/2 x m1

  10. A Little Queuing Theory:Average Wait Time • Calculating average wait time in queue Tq • If something at server, it takes to complete on average m1(z) • Chance server is busy = u; average delay is u x m1(z) • All customers in line must complete; each avg Tser Tq = uxm1(z) + Lq x Ts er= 1/2 x ux Tser x (1 + C) + Lq x Ts er Tq = 1/2 x uxTs er x (1 + C) + r x Tq x Ts er Tq = 1/2 x uxTs er x (1 + C) + u x TqTqx (1 – u) = Ts er x u x (1 + C) /2Tq = Ts er x u x (1 + C) / (2 x (1 – u)) • Notation: r average number of arriving customers/secondTser average time to service a customeru server utilization (0..1): u = r x TserTq average time/customer in queueLq average length of queue:Lq= r x Tq

  11. A Little Queuing Theory: M/G/1 and M/M/1 • Assumptions so far: • System in equilibrium • Time between two successive arrivals in line are random • Server can start on next customer immediately after prior finishes • No limit to the queue: works First-In-First-Out • Afterward, all customers in line must complete; each avg Tser • Described “memoryless” or Markovian request arrival (M for C=1 exponentially random), General service distribution (no restrictions), 1 server: M/G/1 queue • When Service times have C = 1, M/M/1 queueTq = Tser x u x (1 + C) /(2 x (1 – u)) = Tser x u / (1 – u) Tser average time to service a customeru server utilization (0..1): u = r x TserTq average time/customer in queue

  12. A Little Queuing Theory: An Example • processor sends 10 x 8KB disk I/Os per second, requests & service exponentially distrib., avg. disk service = 20 ms • On average, how utilized is the disk? • What is the number of requests in the queue? • What is the average time spent in the queue? • What is the average response time for a disk request? • Notation: r average number of arriving customers/second = 10Tser average time to service a customer = 20 ms (0.02s)u server utilization (0..1): u = r x Tser= 10/s x .02s = 0.2Tq average time/customer in queue = Tser x u / (1 – u) = 20 x 0.2/(1-0.2) = 20 x 0.25 = 5 ms (0 .005s)Tsys average time/customer in system: Tsys =Tq +Tser= 25 msLq average length of queue:Lq= r x Tq= 10/s x .005s = 0.05 requests in queueLsys average # tasks in system: Lsys = r x Tsys = 10/s x .025s = 0.25

  13. A Little Queuing Theory: Another Example • processor sends 20 x 8KB disk I/Os per sec, requests & service exponentially distrib., avg. disk service = 12 ms • On average, how utilized is the disk? • What is the number of requests in the queue? • What is the average time a spent in the queue? • What is the average response time for a disk request? • Notation: r average number of arriving customers/second= 20Tser average time to service a customer= 12 msu server utilization (0..1): u = r x Tser= 20/s x .012s = 0.24Tq average time/customer in queue = Ts er x u / (1 – u) = 12 x 0.24/(1-0.24) = 12 x 0.32 = 3.8 msTsys average time/customer in system: Tsys =Tq +Tser= 15.8 msLq average length of queue:Lq= r x Tq= 20/s x .0038s = 0.076 requests in queueLsys average # tasks in system : Lsys = r x Tsys = 20/s x .016s = 0.32

  14. A Little Queuing Theory:Yet Another Example • Suppose processor sends 10 x 8KB disk I/Os per second, squared coef. var.(C) = 1.5, avg. disk service time = 20 ms • On average, how utilized is the disk? • What is the number of requests in the queue? • What is the average time a spent in the queue? • What is the average response time for a disk request? • Notation: r average number of arriving customers/second= 10Tser average time to service a customer= 20 msu server utilization (0..1): u = r x Tser= 10/s x .02s = 0.2Tq average time/customer in queue = Tser x u x (1 + C) /(2 x (1 – u)) = 20 x 0.2(2.5)/2(1 – 0.2) = 20 x 0.32 = 6.25 msTsys average time/customer in system: Tsys = Tq +Tser= 26 msLq average length of queue:Lq= r x Tq= 10/s x .006s = 0.06 requests in queueLsys average # tasks in system :Lsys = r x Tsys = 10/s x .026s = 0.26

  15. Manufacturing Advantages of Disk Arrays Disk Product Families Conventional: 4 disk designs 14” 3.5” 5.25” 10” High End Low End Disk Array: 1 disk design 3.5”

  16. Replace Small # of Large Disks with Large # of Small Disks! (1988 Disks) IBM 3390 (K) 20 GBytes 97 cu. ft. 3 KW 15 MB/s 600 I/Os/s 250 KHrs $250K IBM 3.5" 0061 320 MBytes 0.1 cu. ft. 11 W 1.5 MB/s 55 I/Os/s 50 KHrs $2K x70 23 GBytes 11 cu. ft. 1 KW 120 MB/s 3900 IOs/s ??? Hrs $150K Data Capacity Volume Power Data Rate I/O Rate MTTF Cost large data and I/O rates high MB per cu. ft., high MB per KW reliability? Disk Arrays have potential for

  17. Array Reliability • Reliability of N disks = Reliability of 1 Disk ÷ N • 50,000 Hours ÷ 70 disks = 700 hours • Disk system MTTF: Drops from 6 years to 1 month! • • Arrays (without redundancy) too unreliable to be useful! Hot spares support reconstruction in parallel with access: very high media availability can be achieved

  18. Redundant Arrays of Disks • Files are "striped" across multiple spindles • Redundancy yields high data availability Disks will fail Contents reconstructed from data redundantly stored in the array Capacity penalty to store it Bandwidth penalty to update Mirroring/Shadowing (high capacity cost) Horizontal Hamming Codes (overkill) Parity & Reed-Solomon Codes Failure Prediction (no capacity overhead!) VaxSimPlus — Technique is controversial Techniques:

  19. Redundant Arrays of DisksRAID 1: Disk Mirroring/Shadowing recovery group • Each disk is fully duplicated onto its "shadow" Very high availability can be achieved • Bandwidth sacrifice on write: Logical write = two physical writes • Reads may be optimized • Most expensive solution: 100% capacity overhead Targeted for high I/O rate , high availability environments

  20. Redundant Arrays of Disks RAID 3: Parity Disk 10010011 11001101 10010011 . . . P logical record 1 0 0 1 0 0 1 1 1 1 0 0 1 1 0 1 1 0 0 1 0 0 1 1 0 0 1 1 0 0 0 0 Striped physical records • Parity computed across recovery group to protect against hard disk failures 33% capacity cost for parity in this configuration wider arrays reduce capacity costs, decrease expected availability, increase reconstruction time • Arms logically synchronized, spindles rotationally synchronized logically a single high capacity, high transfer rate disk Targeted for high bandwidth applications: Scientific, Image Processing

  21. Redundant Arrays of Disks RAID 5+: High I/O Rate Parity Increasing Logical Disk Addresses D0 D1 D2 D3 P A logical write becomes four physical I/Os Independent writes possible because of interleaved parity Reed-Solomon Codes ("Q") for protection during reconstruction D4 D5 D6 P D7 D8 D9 P D10 D11 D12 P D13 D14 D15 Stripe P D16 D17 D18 D19 Targeted for mixed applications Stripe Unit D20 D21 D22 D23 P . . . . . . . . . . . . . . . Disk Columns

  22. Problems of Disk Arrays: Small Writes RAID-5: Small Write Algorithm 1 Logical Write = 2 Physical Reads + 2 Physical Writes D0 D1 D2 D0' D3 P old data new data old parity (1. Read) (2. Read) XOR + + XOR (3. Write) (4. Write) D0' D1 D2 D3 P'

  23. Subsystem Organization array controller host single board disk controller host adapter manages interface to host, DMA single board disk controller control, buffering, parity logic single board disk controller physical device control single board disk controller striping software off-loaded from host to array controller no applications modifications no reduction of host performance often piggy-backed in small format devices

  24. System Availability: Orthogonal RAIDs Array Controller String Controller . . . String Controller . . . String Controller . . . String Controller . . . String Controller . . . String Controller . . . Data Recovery Group: unit of data redundancy Redundant Support Components: fans, power supplies, controller, cables End to End Data Integrity: internal parity protected data paths

  25. System-Level Availability host host Fully dual redundant I/O Controller I/O Controller Array Controller Array Controller . . . . . . . . . Goal: No Single Points of Failure . . . . . . . . . with duplicated paths, higher performance can be obtained when there are no failures Recovery Group

  26. Review: Storage System Issues • Historical Context of Storage I/O • Secondary and Tertiary Storage Devices • Storage I/O Performance Measures • Processor Interface Issues • A Little Queuing Theory • Redundant Arrarys of Inexpensive Disks (RAID) • I/O Buses • ABCs of UNIX File Systems • I/O Benchmarks • Comparing UNIX File System Performance

  27. CS 252 Administrivia • Upcoming schedule of project events in CS 252 • Friday Nov 12: finish I/O? Start multiprocessing/networking • Remaining 3 lectures before Thanksgiving: multiprocessing • Wednesday Dec 1: Midterm I • Friday Dec 3: Esoteric computation. • Quantum/DNA/Nano computing • Next week: Midproject meetings. Tuesday? (Sharad?) • Tue/Wed Dec 7/8 for oral reports? • Friday Dec 10: project reports due.Get moving!!!

  28. Processor Input Control Memory Datapath Output What is a bus? A Bus Is: • shared communication link • single set of wires used to connect multiple subsystems • A Bus is also a fundamental tool for composing large, complex systems • systematic means of abstraction

  29. Buses

  30. Memory Processer Advantages of Buses • Versatility: • New devices can be added easily • Peripherals can be moved between computersystems that use the same bus standard • Low Cost: • A single set of wires is shared in multiple ways I/O Device I/O Device I/O Device

  31. Memory Processer Disadvantage of Buses • It creates a communication bottleneck • The bandwidth of that bus can limit the maximum I/O throughput • The maximum bus speed is largely limited by: • The length of the bus • The number of devices on the bus • The need to support a range of devices with: • Widely varying latencies • Widely varying data transfer rates I/O Device I/O Device I/O Device

  32. General Organization of a Bus Control Lines Data Lines • Control lines: • Signal requests and acknowledgments • Indicate what type of information is on the data lines • Data lines carry information between the source and the destination: • Data and Addresses • Complex commands

  33. Master versus Slave Master issues command Bus Master Bus Slave Data can go either way • A bus transaction includes two parts: • Issuing the command (and address) – request • Transferring the data – action • Master is the one who starts the bus transaction by: • issuing the command (and address) • Slave is the one who responds to the address by: • Sending data to the master if the master ask for data • Receiving data from the master if the master wants to send data

  34. DMA (Direct Memory Access)? • Typical I/O devices must transfer large amounts of data to memory of processor: • Disk must transfer complete block (4K? 16K?) • Large packets from network • Regions of frame buffer • DMA gives external device ability to write memory directly: much lower overhead than having processor request one word at a time. • Processor (or at least memory system) acts like slave • Issue: Cache coherence: • What if I/O devices write data that is currently in processor Cache? • The processor may never see new data! • Solutions: • Flush cache on every I/O operation (expensive) • Have hardware invalidate cache lines (remember “Coherence” cache misses?)

  35. Types of Buses • Processor-Memory Bus (design specific) • Short and high speed • Only need to match the memory system • Maximize memory-to-processor bandwidth • Connects directly to the processor • Optimized for cache block transfers • I/O Bus (industry standard) • Usually is lengthy and slower • Need to match a wide range of I/O devices • Connects to the processor-memory bus or backplane bus • Backplane Bus (standard or proprietary) • Backplane: an interconnection structure within the chassis • Allow processors, memory, and I/O devices to coexist • Cost advantage: one bus for all components

  36. Example: Pentium System Organization Processor/Memory Bus PCI Bus I/O Busses

  37. A Computer System with One Bus: Backplane Bus Backplane Bus Processor Memory • A single bus (the backplane bus) is used for: • Processor to memory communication • Communication between I/O devices and memory • Advantages: Simple and low cost • Disadvantages: slow and the bus can become a major bottleneck • Example: IBM PC - AT I/O Devices

  38. Processor Memory Bus Processor Memory Bus Adaptor Bus Adaptor Bus Adaptor I/O Bus I/O Bus I/O Bus A Two-Bus System • I/O buses tap into the processor-memory bus via bus adaptors: • Processor-memory bus: mainly for processor-memory traffic • I/O buses: provide expansion slots for I/O devices • Apple Macintosh-II • NuBus: Processor, memory, and a few selected I/O devices • SCCI Bus: the rest of the I/O devices

  39. Processor Memory Bus Processor Memory Bus Adaptor Bus Adaptor I/O Bus Backplane Bus Bus Adaptor I/O Bus A Three-Bus System • A small number of backplane buses tap into the processor-memory bus • Processor-memory bus is only used for processor-memory traffic • I/O buses are connected to the backplane bus • Advantage: loading on the processor bus is greatly reduced

  40. Processor Memory North/South Bridge architectures: separate buses Processor Memory Bus “backside cache” • Separate sets of pins for different functions • Memory bus • Caches • Graphics bus (for fast frame buffer) • I/O buses are connected to the backplane bus • Advantage: • Buses can run at different speeds • Much less overall loading! Bus Adaptor I/O Bus Backplane Bus Bus Adaptor I/O Bus

  41. What defines a bus? Transaction Protocol Timing and Signaling Specification Bunch of Wires Electrical Specification Physical / Mechanical Characterisics – the connectors

  42. Synchronous and Asynchronous Bus • Synchronous Bus: • Includes a clock in the control lines • A fixed protocol for communication that is relative to the clock • Advantage: involves very little logic and can run very fast • Disadvantages: • Every device on the bus must run at the same clock rate • To avoid clock skew, they cannot be long if they are fast • Asynchronous Bus: • It is not clocked • It can accommodate a wide range of devices • It can be lengthened without worrying about clock skew • It requires a handshaking protocol

  43. Busses so far Master Slave ° ° ° Control Lines Bus Master: has ability to control the bus, initiates transaction Bus Slave: module activated by the transaction Bus Communication Protocol: specification of sequence of events and timing requirements in transferring information. Asynchronous Bus Transfers: control lines (req, ack) serve to orchestrate sequencing. Synchronous Bus Transfers: sequence relative to common clock. Address Lines Data Lines

  44. Bus Transaction • Arbitration: Who gets the bus • Request: What do we want to do • Action: What happens in response

  45. Arbitration: Obtaining Access to the Bus Control: Master initiates requests Bus Master Bus Slave Data can go either way • One of the most important issues in bus design: • How is the bus reserved by a device that wishes to use it? • Chaos is avoided by a master-slave arrangement: • Only the bus master can control access to the bus: It initiates and controls all bus requests • A slave responds to read and write requests • The simplest system: • Processor is the only bus master • All bus requests must be controlled by the processor • Major drawback: the processor is involved in every transaction

  46. Multiple Potential Bus Masters: the Need for Arbitration • Bus arbitration scheme: • A bus master wanting to use the bus asserts the bus request • A bus master cannot use the bus until its request is granted • A bus master must signal to the arbiter after finish using the bus • Bus arbitration schemes usually try to balance two factors: • Bus priority: the highest priority device should be serviced first • Fairness: Even the lowest priority device should never be completely locked out from the bus • Bus arbitration schemes can be divided into four broad classes: • Daisy chain arbitration • Centralized, parallel arbitration • Distributed arbitration by self-selection: each device wanting the bus places a code indicating its identity on the bus. • Distributed arbitration by collision detection: Each device just “goes for it”. Problems found after the fact.

  47. The Daisy Chain Bus Arbitrations Scheme Device 1 Highest Priority Device N Lowest Priority Device 2 • Advantage: simple • Disadvantages: • Cannot assure fairness: A low-priority device may be locked out indefinitely • The use of the daisy chain grant signal also limits the bus speed Grant Grant Grant Release Bus Arbiter Request wired-OR

  48. Centralized Parallel Arbitration Device 1 Device N Device 2 • Used in essentially all processor-memory busses and in high-speed I/O busses Req Grant Bus Arbiter

  49. Simplest bus paradigm • All agents operate synchronously • All can source / sink data at same rate • => simple protocol • just manage the source and target

  50. Simple Synchronous Protocol BReq • Even memory busses are more complex than this • memory (slave) may take time to respond • it may need to control data rate BG R/W Address Cmd+Addr Data1 Data2 Data

More Related