Memory Coherence in Shared Virtual Memory Systems

1. Memory Coherence in Shared Virtual Memory Systems Kai Li, Paul Hudak ACM Transactions on Computer Systems, Vol. 7, No. 4, November 1989

2. Abstract • The memory coherence problem in designing and implementing a shared virtual memory on loosely coupled multiprocessors is studied in depth. • Two classes of algorithms, centralized and distributed, for solving the problem are presented. Po-Sen Wang

3. Outline • Introduction • Shared Virtual Memory • Memory Coherence Problem • Centralized Manager Algorithm • Distributed Manager Algorithms • Experiment • Conclusions Po-Sen Wang

4. Introduction • The shared virtual memory described in this paper this paper provides a virtual address space that is shared among all processors in a loosely coupled distributed-memory multiprocessor system. • The shared virtual memory provides a natural and efficient form of process migration between processors in a distributed system. Po-Sen Wang

5. Shared Virtual Memory Po-Sen Wang

6. Shared Virtual Memory (cont.) • A shared virtual memory address space is partitioned into pages. • read-only: It can have copies residing in the physical memories of many processors at the same time. • write: It can reside in only one processor’s physical memory. Po-Sen Wang

7. Shared Virtual Memory (cont.) • When page fault happens, the memory mapping manager retrieves the page from either disk or the memory of another processor. • This paper discusses both centralized manager algorithms and distributed manager algorithms, and shows that can retrieve pages efficiently while keeping the memory coherent. Po-Sen Wang

8. Memory Coherence Problem • There are two design choices that greatly influence the implementation of a shared virtual memory: • The granularity of the memory units • The strategy for maintaining coherence Po-Sen Wang

9. Memory Coherence Problem (cont.) Po-Sen Wang

10. Page Synchronization Method • Invalidation • If a processor Q has a write fault to a page p • Invalidates all copies of p • Changes the access of p to write • Moves a copy of p to Q if Q does not have one already • Returns to the faulting instruction • If a processor Q has a read fault to a page p • Changes the access of p to read on the processor that has write access to p • Moves a copy of p to Q and sets the access of p to read • Returns to the faulting instruction Po-Sen Wang

11. Page Synchronization Method (cont.) • Write-broadcast • A processor treats a read fault just as it does in the invalidation approach. • If a processor has a write fault, the fault handler then • Writes to all copies of the page • Returns to the faulting instruction Po-Sen Wang

12. Page Ownership Strategy • Fixed ownership: • A page is always owned by the same processor. • Dynamic page ownership: • It can be subdivided into two classes: centralized and distributed. • Distributed managers can be further classified as either fixed or dynamic, referring to the distribution of ownership data. Po-Sen Wang

13. Ask page p Ask page p Send page p Send page p Forward request Centralized Manager Algorithm • Read fault manager P0 P1 P2 P2’s PTable P0’s PTable P1’s PTable access: access: read access: read read Info lock: lock: lock: Owner: true P2 true true copyset: copyset: copyset: P0 P1 Po-Sen Wang

14. Ask write p Send page p and copyset Centralized Manager Algorithm (cont.) • Write fault manager P0 P1 P2 P2’s PTable P0’s PTable P1’s PTable access: access: write nil read nil access: nil read read Info lock: lock: lock: Owner: P2 P0 true true copyset: copyset: P0 P1 copyset: P0 P1 Po-Sen Wang

15. Distributed Manager Algorithms • A Fixed Distributed Manager Algorithm • Every processor is given a predetermined subset of the pages to manage. • When a fault occurs on page p, the faulting processor asks processor H(p) where the true page owner is, and then proceeds as in the centralized manager algorithm. Po-Sen Wang

16. Forward p Ask p Send page p and copyset Distributed Manager Algorithms (cont.) • A Dynamic Distributed Manager Algorithm • Read fault P0 P1 P2 P0’s PTable P1’s PTable P2’s PTable probOwner: probOwner : P0 P0 probOwner : P1 P2 P2 P0 access : access : read access : read lock : lock : true true lock : true copyset : P2 copyset: copyset: P2 Po-Sen Wang

17. Send page p and copyset Ask p Forward p Distributed Manager Algorithms (cont.) • A Dynamic Distributed Manager Algorithm • Write fault P0 P1 P2 P0’s PTable P1’s PTable P2’s PTable probOwner: probOwner : probOwner : P2 P0 P1 P1 P0 access : access : read nil write access : read nil lock : lock : true true lock : true copyset : P2 copyset: P2 copyset: P2 Po-Sen Wang

18. Experiment • We have implemented a prototype shared virtual memory system called IVY (Integrated shared Virtual memory at Yale). • It is implemented on top of a modified Aegis operating system of the Apollo DOMAIN computer system. • IVY can be used to run parallel programs on any number of processors on an Apollo ring network. Po-Sen Wang

19. Experiment (cont.) Po-Sen Wang

20. Conclusions • The centralized manager algorithm is easy to implement, but it may have a traffic bottleneck at the central manager when there are many page faults. • The fixed distributed manager algorithm alleviates the bottleneck, but, on average, a processor still needs to spend about two messages to locate an owner. • In general, dynamic distributed manager algorithms perform better than other methods. Po-Sen Wang

21. Thank you Po-Sen Wang