MPI and OpenMP

1. MPI and OpenMP

2. How to get MPI and How to Install MPI? • http://www-unix.mcs.anl.gov/mpi/ • mpich2-doc-install.pdf // installation guide • mpich2-doc-user.pdf // user guide

3. Outline • Message-passing model • Message Passing Interface (MPI) • Coding MPI programs • Compiling MPI programs • Running MPI programs • Benchmarking MPI programs • OpenMP

4. Message-passing Model

5. Processes • Number is specified at start-up time • Remains constant throughout execution of program • All execute same program • Each has unique ID number • Alternately performs computations and communicates

6. Not satisfied Circuit Satisfiability 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1

7. Solution Method • Circuit satisfiability is NP-complete • No known algorithms to solve in polynomial time • We seek all solutions • We find through exhaustive search • 16 inputs  65,536 combinations to test

8. Partitioning: Functional Decomposition • Embarrassingly parallel: No channels between tasks

9. Agglomeration and Mapping • Properties of parallel algorithm • Fixed number of tasks • No communications between tasks • Time needed per task is variable • Consult mapping strategy decision tree • Map tasks to processors in a cyclic fashion

10. Cyclic (interleaved) Allocation • Assume p processes • Each process gets every pth piece of work • Example: 5 processes and 12 pieces of work • P0: 0, 5, 10 • P1: 1, 6, 11 • P2: 2, 7 • P3: 3, 8 • P4: 4, 9

11. Summary of Program Design • Program will consider all 65,536 combinations of 16 boolean inputs • Combinations allocated in cyclic fashion to processes • Each process examines each of its combinations • If it finds a satisfiable combination, it will print it

12. Include Files • MPI header file #include <mpi.h> #include <stdio.h> • Standard I/O header file

13. Local Variables int main (int argc, char *argv[]) { int i; int id; /* Process rank */ int p; /* Number of processes */ void check_circuit (int, int); • Include argc and argv: they are needed to initialize MPI • One copy of every variable for each process running this program

14. Initialize MPI • First MPI function called by each process • Not necessarily first executable statement • Allows system to do any necessary setup MPI_Init (&argc, &argv);

15. Communicators • Communicator: opaque object that provides message-passing environment for processes • MPI_COMM_WORLD • Default communicator • Includes all processes • Possible to create new communicators • Will do this in Chapters 8 and 9

16. Communicator Name Communicator Processes Ranks Communicator MPI_COMM_WORLD 0 5 2 1 4 3

17. Determine Number of Processes • First argument is communicator • Number of processes returned through second argument MPI_Comm_size (MPI_COMM_WORLD, &p);

18. Determine Process Rank • First argument is communicator • Process rank (in range 0, 1, …, p-1) returned through second argument MPI_Comm_rank (MPI_COMM_WORLD, &id);

19. 2 0 5 4 3 1 id id id id id id 6 6 6 6 6 6 p p p p p p Replication of Automatic Variables

20. What about External Variables? int total; int main (int argc, char *argv[]) { int i; int id; int p; … • Where is variable total stored?

21. Cyclic Allocation of Work for (i = id; i < 65536; i += p) check_circuit (id, i); • Parallelism is outside function check_circuit • It can be an ordinary, sequential function

22. Shutting Down MPI • Call after all other MPI library calls • Allows system to free up MPI resources MPI_Finalize();

23. Only process 0 will get the result Our Call to MPI_Reduce() MPI_Reduce (&count, &global_count, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); if (!id) printf ("There are %d different solutions\n", global_count);

24. Benchmarking the Program • MPI_Barrier barrier synchronization • MPI_Wtick  timer resolution • MPI_Wtime  current time

25. How to form a Ring? • Now to form ring of systems first of all one has to execute following command. [nayan@MPI_system1 ~]$ mpd & [1] 9152   Which make MPI_system1 as master system of ring. [nayan@MPI_system1 ~]$ mpdtrace –l MPI_system1_32958 (172.16.1.1)

26. How to form a Ring? (cont’d) • Then run following command in terminal of all other system [nayan@MPI_system1 ~]$ mpd –h MPI_system1 –p 32958 & Port number of Master system Host name of Master system

27. How to kill a Ring? • And to kill ring run following command on master system.  [nayan@MPI_system1 ~]$ mpdallexit

28. Compiling MPI Programs • to compile and execute above program follow the following steps • first of all compile sat1.c by executing following command on master system.  [nayan@MPI_system1 ~]$ mpicc -0 sat1.out sat1.c here “mpicc” is mpi command to compile sat1.c file and sat1.out is output file.

29. Running MPI Programs • Now to run this output file type following command in master system  [nayan@MPI_system1 ~]$ mpiexec -n 1 ./sat1.out

30. Benchmarking Results

31. OpenMP • OpenMP: An application programming interface (API) for parallel programming on multiprocessors • Compiler directives • Library of support functions • OpenMP works in conjunction with Fortran, C, or C++

32. Shared-memory Model Processors interact and synchronize with each other through shared variables.

33. Fork/Join Parallelism • Initially only master thread is active • Master thread executes sequential code • Fork: Master thread creates or awakens additional threads to execute parallel code • Join: At end of parallel code created threads die or are suspended

34. Fork/Join Parallelism

35. Shared-memory Model vs.Message-passing Model • Shared-memory model • Number active threads 1 at start and finish of program, changes dynamically during execution • Message-passing model • All processes active throughout execution of program

36. Parallel for Loops • C programs often express data-parallel operations as for loops for (i = first; i < size; i += prime) marked[i] = 1; • OpenMP makes it easy to indicate when the iterations of a loop may execute in parallel • Compiler takes care of generating code that forks/joins threads and allocates the iterations to threads

37. Pragmas • Pragma: a compiler directive in C or C++ • Stands for “pragmatic information” • A way for the programmer to communicate with the compiler • Compiler free to ignore pragmas • Syntax: #pragma omp<rest of pragma>

38. Parallel for Pragma • Format: #pragma omp parallel for for (i = 0; i < N; i++) a[i] = b[i] + c[i]; • Compiler must be able to verify the run-time system will have information it needs to schedule loop iterations

39. Function omp_get_num_procs • Returns number of physical processors available for use by the parallel program int omp_get_num_procs (void)

40. Function omp_set_num_threads • Uses the parameter value to set the number of threads to be active in parallel sections of code • May be called at multiple points in a program void omp_set_num_threads (int t)

41. Comparison

42. C+MPI vs. C+MPI+OpenMP C + MPI C + MPI + OpenMP

43. Benchmarking Results

44. Example Programm

45. How to run OpenMP programm $ export OMP_NUM_THREAD=2 $ gcc –fopenmp filename.c $ ./a.out

46. For more information • http://www.cs.ccu.edu.tw/~naiwei/cs5635/cs5635.html • http://www.nersc.gov/nusers/help/tutorials/mpi/intro/print.php • http://www-unix.mcs.anl.gov/mpi/tutorial/mpiintro/ppframe.htm • http://www.mhpcc.edu/training/workshop/mpi/MAIN.html • http://www.openmp.org