Distributed Shared Memory Systems and Programming

1. Distributed Shared Memory Systems and Programming By: Kenzie MacNeil Adapted from Parallel Programming Techniques and Applications using networked workstations and parallel computers by Barry Wilkinson and Michael Allen, and

2. Distributed Shared Memory Systems • Shared memory programming model on al cluster • Has physically distributed and separate memory • Programming Viewpoint: • Memory is grouped together and sharable between processes • Known as Distributed Shared Memory (DSM)

3. Distributed Shared Memory Systems • Can be achieved by software or hardware • Software: • Easy to use on clusters • Inferior to using explicit message passing on the same cluster • Utilizes the same techniques as true shared memory systems (Chapter 8)

4. Distributed Shared Memory • Shared memory programming is generally more convenient than message passing • Data can be accessed by individual processors without explicitly sending data • Shared data has to be controlled • Locks or other means • Both message passing and shared memory often require synchronization

5. Distributed Shared Memory • Distributed Shared Memory is a group of interconnected computers appearing to have a sing memory with a single address space • Each computer having its own memory which is physically distributed • Any memory location can be accessed by any processor in the cluster • Regardless of the memory residing locally

6. Distributed Shared Memory

7. Advantages of DMS • Normal shared memory programming techniques can be used • Easily scalable, compared to traditional bus-connected shared memory multiprocessors • Message passing is hidden from the user • Can handle complex and large data bases without replication or sending the data to processes

8. Disadvantages of DMS • Lower performance than true shared memory multiprocessor systems • Must provide for protection against simultaneous access to shared data • Locks, etc. • Little programmer control over actual messages being generated • Incur performance penalties when compared to message passing routines on a cluster

9. Hardware DSM Systems • Special network interfaces and cache coherence circuits are required • Several interfaces that support shared memory operations • Higher level of performance • More expensive

10. Software DSM Systems • Requires no hardware changes • Preformed by software routines • Software layer added between the operating system and the applications • Kernel may or may not be modified • Software layer can be • Page based • Shared variable based • Object based

11. Page Based DMS • Existing virtual memory is used to instigate movement of data between computer • Occurs when page referenced does not reside locally • Referred to as virtual shared memory system • Page based systems include: • The first DMS system by Li(1986), TreadMarks (1996), Locust (1998)

12. Page Based DSM System

13. Page Based DMS Disadvantages • Size of the unit of the data, a page, can be too big • More than the specific data is usually referenced • Leads to longer messages • Not portable, because they are tied to a particular virtual memory hardware and software • False sharing effects appear at the page level • Situation in which different parts of a page are required by different processors without any actual sharing of information, but each page must be shared by each process to access different parts

14. Shared Variable DMS • Only variables declared as shared are transferred • Transferred on demand • Paging mechanism is not used • Software routines perform the actions • Shared Variable DMS approach includes: • Munin (1990), JIAJIA (1999), Adsmith (1996)

15. Object Based DMS • Shared data is embodied in objects • Includes data items and procedures/methods • Methods used to access data • Similar to shared variable approach, even considered an extension • Easily implemented in OO languages

16. Managing Shared Data • Many ways a processor can be given access to shared data • Simplest is the use of a central server • Responsible for all read write operations on shared data • Requests sent to this server • Occurs sequentially on the server • Implements a single reader/ single writer policy

17. Managing Shared Data • Single reader/writer policy incurs bottleneck • Additional servers can be added to relieve this bottleneck by sharing variables • However multiple copies of data is preferable • Allows simultaneous access to the data by different processors • Coherence policy must be used to maintain these copies

18. Multiple Reader / Single Writer • Allows multiple processors to read shared data • Which can be achieved by replicating data • Allows only one processor, the owner, to alter data at any instant • When an owner alters data two policies are available: • Update policy • Invalidate policy

19. Multiple Reader/Single Writer Policy • Update policy • Utilizes broadcast • All copies are altered to reflect broadcast message • Invalidate policy • All unaltered copies of the data are flagged as invalid • Requires a processor to make a request from the owner • Any copies of the data that are not accessed remain invalid • Both policies are needed to be reliable

20. Multiple Reader/Single Writer Policy • Page based approach • Complete page, which holds the variable, is transferred • A variable stored on a page which is not shared will be moved or invalidated • Protocols offered by applications like TreadMarks for dual writing to a single page

21. Achieving Consistent Memory in DSM • Memory consistency addresses when the current value of a shared variable is seen by other processors • Various models are available: • Strict Consistency • Sequential Consistency • Relaxed Consistency • Weak consistency • Release Consistency • Lazy Release Consistency

22. Strict Consistency • Variable is obtained from the most recent write to the shared variable • As soon as a variable is altered all other processors are informed • Can be done by update or invalidity • Disadvantage is the large number of messages and changes are not instantaneous • Relaxed memory consistency, writes are delayed to reduce message passing

23. Strict Consistency

24. Sequential and Weak Consistency • Sequential consistency, result of any execution same as an interleaving of individual programs • Weak consistency, synchronized operations are used by the programmer to enforce sequential consistency • Any accesses to shared data can be controlled with synchronized operations • Locks, etc

25. Release Consistency • Extension of weak consistency • Specified synchronization operation • Acquire operation, used before a shared variable or variables are to be read • Release operations, used after the shared variable or variable have been altered • Acquire is performed with a lock operation • Release is performed with an unlock operation

26. Release Consistency

27. Lazy Release Consistency • Version of release consistency • Update is only done at the time of acquire rather than at release • Generates fewer messages that release consistency

28. Lazy Release Consistency

29. Distributed Shared Memory Programming Primitives • Four fundamental and necessary operations of shared memory programming: • Process/thread creations and termination • Shared data creation • Mutual exclusion synchronization, controlled access to shared data • Process/thread and event synchronization • Typically provided by user-level library calls

30. Process Creation • Set of routines are defined by DSM systems • Such as Adsmith and TreadMarks • Used to start new process if process creation is supported • dsm_spawn(filename, num_processes);

31. Shared Data Creation • Routine is necessary to declare shared data • dsm_shared(&x); or shared int x; • Dynamically creates memory space for shared data in the manner of a C malloc • After memory space can be discarded

32. Shared Data Access • Various forms of data access are provided depending on the memory consistency used • Some systems provide efficient routines for difference classes of accesses • Adsmith provides three types of accesses: • Ordinary Accesse • Synchronization Access • Non-Synchronization Access

33. Synchronization Accesses • Two principle forms: • Global synchronization and process-process pair synchronization • Global is usually done through barrier routines • Process-process pair can be done by the same routine or separate routines through simple synchronous send/receive routines • DSM systems could also provide their own routines

34. Overlapping Computations with Communications • Can be provided by starting a nonblocking communication before it results are needed • Called a prefetch routine • Program continues execution after the prefetch has been called and while the data is being fetched • Could even be done speculatively • Special mechanism must be in place to handle memory exceptions • Similar to speculative load mechanism used in advanced processors that overlap memory operations with program execution

35. Distributed Shared Memory Programming • DSM programming on a cluster uses the same concepts as shared memory programming on a shared memory multiprocessor system • Uses user level library routines or methods • Message passing is hidden from the user

36. Basic Shared-Variable Implementation • Simplest DSM implementation is to use a shared variable approach with user level DSM library routines • Sitting on top of an existing message passing systems, such as MPI • Routines can be embodied into classes and methods • The routines could send messages to a central location that is responsible for the shared variables

37. Simple DSM System using a Centralized Server Single reader/writer protocol

38. Basic Shared-Variable Implementation • A simple DSM system using a centralized server can easily result in a bottleneck • One method to reduce this bottleneck is to have multiple servers running on different processors • Each server responsible for specific shared variables • This is a single reader / single writer protocol

39. Simple DSM System using Multiple Servers

40. Basic Shared-Variable Implementation • Also can provide multiple reader capability • A specific server is responsible for the shared variable • Other local copies are invalidated

41. Simple DSM System using Multiple Servers and Multiple Reader Policy

42. Overlapping Data Groups • Existing interconnections structure • Access patterns of the application • Static overlapping • Defined by the programmer prior to execution • Shared variables can migrate according to usage

43. Symmetrical Multiprocessor System with Overlapping Data Regions

44. Simple DSM System using Multiple Servers and Multiple Reader Policy

45. Questions or Comments?