1 / 84

Prof. Chris Carothers Computer Science Department Lally 306 [Office Hrs: Wed, 11a.m – 1p.m]

CSCI-6964: High Performance Parallel & Distributed Computing (HPDC) AE 216, Mon/Thurs. 2 – 3:20 p.m Memory/Caching Overview. Prof. Chris Carothers Computer Science Department Lally 306 [Office Hrs: Wed, 11a.m – 1p.m] chrisc@cs.rpi.edu www.cs.rpi.edu/~chrisc/COURSES/HPDC/SPRING-2008.

cvirginia
Download Presentation

Prof. Chris Carothers Computer Science Department Lally 306 [Office Hrs: Wed, 11a.m – 1p.m]

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSCI-6964: High Performance Parallel & Distributed Computing (HPDC)AE 216, Mon/Thurs. 2 – 3:20 p.mMemory/Caching Overview Prof. Chris Carothers Computer Science Department Lally 306 [Office Hrs: Wed, 11a.m – 1p.m] chrisc@cs.rpi.edu www.cs.rpi.edu/~chrisc/COURSES/HPDC/SPRING-2008 HPDC Spring 2008 - Parallel Programming Platforms

  2. Memory Technologies: Speed vs. Cost (1997) Access Time: the length of time it takes to get a value from memory, given an address. HPDC Spring 2008 - Parallel Programming Platforms

  3. Memory Technologies: Speed vs. Cost (2004) Observe: access time not changing much over the last 7 years, but unit cost per capacity has changed dramatically HPDC Spring 2008 - Parallel Programming Platforms

  4. Performance and Memory • SRAM is fast, but too expensive (we want large memories!). • Using only SRAM (enough of it) would mean that the memory ends up costing more than everything else combined! HPDC Spring 2008 - Parallel Programming Platforms

  5. Caching • The idea is to use a small amount of fast memory near the processor (in a cache). • The cache hold frequently needed memory locations. • when an instruction references a memory location, we want that value to be in the cache! HPDC Spring 2008 - Parallel Programming Platforms

  6. Principles of Locality time Temporal: if a memory location is referenced, it is likely that it will be referenced again in the near future. Spatial: if a memory location is referenced, it is likely that nearby items will be referenced in the near future. space HPDC Spring 2008 - Parallel Programming Platforms

  7. Programs and Locality Programs tend to exhibit a great deal of locality in memory accesses. • array, structure/record access • subroutines (instructions are near each other) • local variables (counters, pointers, etc) are often referenced many times. HPDC Spring 2008 - Parallel Programming Platforms

  8. Memory Hierarchy The general idea is to build a hierarchy: • at the top is a small, fast memory that is close to the processor. • in the middle are larger, slower memories. • At the bottom is massive memory with very slow access time. HPDC Spring 2008 - Parallel Programming Platforms

  9. C P U I n c r e a s i n g d i s t a n c e L e v e l 1 f r o m t h e C P U i n a c c e s s t i m e L e v e l 2 L e v e l s i n t h e m e m o r y h i e r a r c h y L e v e l n S i z e o f t h e m e m o r y a t e a c h l e v e l Figure 7.3 HPDC Spring 2008 - Parallel Programming Platforms

  10. Cache and Main Memory • For now we will focus on a 2 level hierarchy: • cache (small, fast memory directly connected to the processor). • main memory (large, slow memory at level 2 in the hierarchy). HPDC Spring 2008 - Parallel Programming Platforms

  11. P r o c e s s o r D a t a a r e t r a n s f e r r e d Memory Hierarchy andData Transfer Transfer of data is done between adjacent levels in the hierarchy only! All access by the processor is to the topmost level. Figure 7.2 HPDC Spring 2008 - Parallel Programming Platforms

  12. Terminology • hit: when the memory location accessed by the processor is in the cache (upper level). • miss: when the memory location accessed by the process is not in the cache. • block:the minimum unit of information transferred between the cache and the main memory. Typically measured in bytes or words. HPDC Spring 2008 - Parallel Programming Platforms

  13. Terminology (cont.) • hit rate: the ratio of hits to total memory accesses. • miss rate: 1 – hit rate • hit time: the time to access an element that is in the cache: • time to find out if it’s in the cache. • time to transfer from cache to processor. HPDC Spring 2008 - Parallel Programming Platforms

  14. Terminology (cont.) • miss penalty: the time to replace a block in the cache with a block from main memory and to deliver deliver the element to the processor. • hit timeis small compared to miss penalty (otherwise we wouldn’t bother with a memory hierarchy!) HPDC Spring 2008 - Parallel Programming Platforms

  15. Simple Cache Model • Assume that the processor accesses memory one word at a time. • A block consists of one word. • When a word is referenced and is not in the cache, it is put in the cache (copied from main memory). HPDC Spring 2008 - Parallel Programming Platforms

  16. Cache Usage • At some point in time the cache holds memory items X1,X2,…Xn-1 • The processor next accesses memory item Xn which is not in the cache. HPDC Spring 2008 - Parallel Programming Platforms

  17. Cache before and after HPDC Spring 2008 - Parallel Programming Platforms

  18. Issues • How do we know if an item is in the cache? • If it is in the cache, how do we know where it is? HPDC Spring 2008 - Parallel Programming Platforms

  19. Direct-Mapped Cache • Each memory location is mapped to a single location in the cache. • there in only one place it can be! • Remember that the cache is smaller than memory, so many memory locations will be mapped to the same location in the cache. HPDC Spring 2008 - Parallel Programming Platforms

  20. Mapping Function • The simplest mapping is based on the LS bits of the address. • For example, all memory locations whose address ends in 000 will be mapped to the same location in the cache. • The requires a cache size of 2n locations (a power of 2). HPDC Spring 2008 - Parallel Programming Platforms

  21. A Direct Mapped Cache Figure 7.5 HPDC Spring 2008 - Parallel Programming Platforms

  22. Who’s in slot000? • We still need a way to find out which of the many possible memory elements is currently in a cache slot. • slot: a location in the cache that can hold a block. • We need to store the address of the item currently using cache slot 000. HPDC Spring 2008 - Parallel Programming Platforms

  23. Tags • We don’t need to store the entire memory location address, just those bits that are not used to determine the slot number (the mapping). • We call these bits the tag. • The tag associated with a cache slot tells who is currently using the slot. HPDC Spring 2008 - Parallel Programming Platforms

  24. 16 word memory, 4 word cache Memory 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 Tags Data HPDC Spring 2008 - Parallel Programming Platforms

  25. Initialization Problem • Initially the cache is empty. • all the bits in the cache (including the tags) will have random values. • After some number of accesses, some of the tags are real and some are still just random junk. • How do we know which cache slots are junk and which really mean something? HPDC Spring 2008 - Parallel Programming Platforms

  26. Valid Bits • Include one more bit with each cache slot that indicates whether the tag is valid or not. • Provide hardware to initialize these bits to 0 (one bit per cache slot). • When checking a cache slot for a specific memory location, ignore the tag if the valid bit is 0. • Change a slot’s valid bit to a 1 when putting something in the slot (from main memory). HPDC Spring 2008 - Parallel Programming Platforms

  27. Revised Cache Memory Valid 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111 Tags Data HPDC Spring 2008 - Parallel Programming Platforms

  28. Simple Simulation • We can simulate the operation of our simple direct-mapped cache by listing a sequence of memory locations that are referenced. • Assume the cache is initialized with all the valid bits set to 0 (to indicate all the slots are empty). HPDC Spring 2008 - Parallel Programming Platforms

  29. Memory Access Sequence HPDC Spring 2008 - Parallel Programming Platforms

  30. Hardware • We need to have hardware that can perform all the operations: • find the right slot given an address (perform the mapping). • check the valid bit. • compare the tag to part of the address HPDC Spring 2008 - Parallel Programming Platforms

  31. A d d r e s s ( s h o w i n g b i t p o s i t i o n s ) 3 1 3 0 1 3 1 2 1 1 2 1 0 B y t e o f f s e t 1 0 2 0 H i t D a t a T a g I n d e x I n d e x V a l i d T a g D a t a 0 1 2 1 0 2 1 1 0 2 2 1 0 2 3 2 0 3 2 HPDC Spring 2008 - Parallel Programming Platforms

  32. Handling a Cache Miss • A miss means the processor must wait until the memory requested is in the cache. • a separate controller handles transferring data between the cache and memory. • In general the processor continuously tries the fetch until it works (until it’s a hit). • continuously means “once per cycle”. • in the meantime the pipeline is stalled! HPDC Spring 2008 - Parallel Programming Platforms

  33. Data vs. Instruction Cache • Obviously nothing other than a stall can happen if we get a miss when fetching the next instruction! • It is possible to execute other instructions while waiting for data (need to detect data hazards), this is called stall on use. • the pipeline stalls only when there are no instructions that can execute without the data. HPDC Spring 2008 - Parallel Programming Platforms

  34. DecStation 3100 Cache • Simple Cache implementation • 64 KB cache (16K words). • 16 bit tags • Direct Mapped • Two caches, one for instructions and the other for data. HPDC Spring 2008 - Parallel Programming Platforms

  35. d d r s s ( w i i i i s ) A e s h o n g b t p o s t o n 3 1 3 0 1 7 1 6 1 5 5 4 3 2 1 0 B y t e 1 6 1 4 o f f s e t H i t D a t a 1 6 b i t s 3 2 b i t s V a l i d T a g D a t a 1 6 K e n t r i e s 1 6 3 2 DecStation 3100 cache HPDC Spring 2008 - Parallel Programming Platforms

  36. Handling Writes • What happens when a store instruction is executed? • what if it’s a hit? • what if it’s a miss? • DecStation 3100 does the following: • don’t bother checking the cache, just write the new value in to the cache! • Also write the word to main memory (called write-through). HPDC Spring 2008 - Parallel Programming Platforms

  37. Write-Through • Always updating main memory on each store instruction can slow things down! • the memory is tied up for a while. • It is possible to set up a write buffer that holds a number of pending writes. • If we also update the cache, it is not likely that we need to worry about getting a memory value from the buffer (but it’s possible!) HPDC Spring 2008 - Parallel Programming Platforms

  38. Write-back • Another scheme for handling writes: • only update the cache. • when the memory location is booted out of the cache (someone else is being put in to the same slot), write the value to memory. HPDC Spring 2008 - Parallel Programming Platforms

  39. Cache Performance For the simple DecStation 3100 cache: HPDC Spring 2008 - Parallel Programming Platforms

  40. Spatial Locality? • So far we’ve only dealt with temporal locality (it we access an item, it is likely we will access it again soon). • What about space (the final frontier)? • In general we make a block hold more than a single word. • Whenever we move data to the cache, we also move it’s neighbors. HPDC Spring 2008 - Parallel Programming Platforms

  41. Blocks and Slots • Each cache slot holds one block. • Given a fixed cache size (number of bytes) as the block size increases, the number of slots must decrease. • Reducing the number of slots in the cache increases the number of memory locations that compete for the same slot. HPDC Spring 2008 - Parallel Programming Platforms

  42. Example multi-word block cache • 4 words/block • we now use a block address to determine the slot mapping. • the block address in this case is the address/4. • on a hit we need to extract a single word (need a multiplexor controlled by the LS 2 address bits). • 64KB data • 16 Bytes/block • 4K slots. HPDC Spring 2008 - Parallel Programming Platforms

  43. A d d r e s s ( s h o w i n g b i t p o s i t i o n s ) 3 1 1 6 1 5 4 3 2 1 0 1 6 1 2 2 B y t e H i t D a t a T a g o f f s e t I n d e x B l o c k o f f s e t 1 6 b i t s 1 2 8 b i t s V D a t a T a g 4 K e n t r i e s 1 6 3 2 3 2 3 2 3 2 M u x 3 2 HPDC Spring 2008 - Parallel Programming Platforms

  44. Performance and Block Size DecStation 3100 cache with block sizes 1 and 4 (words). HPDC Spring 2008 - Parallel Programming Platforms

  45. Is bigger always better? • Eventually increasing the block size will mean that the competition for cache slots is too high • miss rate will increase. • Consider the extreme case: the entire cache is a single block! HPDC Spring 2008 - Parallel Programming Platforms

  46. 4 0 % 3 5 % 3 0 % 2 5 % e t a r s 2 0 % s i M 1 5 % 1 0 % 5 % 0 % 4 1 6 6 4 2 5 6 B l o c k s i z e ( b y t e s ) 1 K B 8 K B 1 6 K B 6 4 K B 2 5 6 K B Miss rate vs. Block Size HPDC Spring 2008 - Parallel Programming Platforms

  47. Block Size and Miss Time • As the block size increases, we need to worry about what happens to the miss time. • The larger a block is, the longer it takes to transfer from main memory to cache. • It is possible to design memory systems with transfer of an entire block at a time, but only for relatively small block sizes (4 words). HPDC Spring 2008 - Parallel Programming Platforms

  48. Example Timings Hypothetical access times: • 1 cycle to send the address • 15 cycles to initiate each access • 1 cycle to transfer each word. • Miss penalty for 4-word wide memory is: 1 + 4x15 + 4x1 = 65 cycles. HPDC Spring 2008 - Parallel Programming Platforms

  49. C P U C P U C P U M u l t i p l e x o r C a c h e C a c h e C a c h e B u s B u s B u s M e m o r y M e m o r y M e m o r y M e m o r y M e m o r y b a n k 0 b a n k 1 b a n k 2 b a n k 3 c . I n t e r l e a v e d m e m o r y o r g a n i z a t i o n b . W i d e m e m o r y o r g a n i z a t i o n M e m o r y a . O n e - w o r d - w i d e m e m o r y o r g a n i z a t i o n Memory Organization Options Improving memory bandwidth…. HPDC Spring 2008 - Parallel Programming Platforms

  50. Improving Cache Performance • Cache performance is based on two factors: • miss rate • depends on both the hardware and on the program being measured (miss rate can vary). • miss penalty • the penalty is dictated by the hardware (the organization of memory and memory access times). HPDC Spring 2008 - Parallel Programming Platforms

More Related