Lecture 2 part i parallel programming models
Sponsored Links
This presentation is the property of its rightful owner.
1 / 42

Lecture 2: Part I Parallel Programming Models PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Lecture 2: Part I Parallel Programming Models. A programming model is what the programmer sees and uses when developing program. Sequential Programming Model. Processor. P. M. code. Memory. os. Parallel Programming.

Download Presentation

Lecture 2: Part I Parallel Programming Models

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Lecture 2: Part IParallel Programming Models

A programming model is what the programmer sees and uses when developing program

Sequential Programming Model







Parallel Programming

  • Definition: PP is the activity of constructing a parallel program from a given algorithm.

  • Interface between algorithms and parallel architectures.

Why Is It So Difficult ?

  • Potentially more complicated than sequential programming

  • Diverse parallel programming models

  • Lack of advanced parallel compiler, debugger, profiler.

  • More people doing sequential programming


Can we run sequential C code on parallel machine to gain speedup ?

Levels of Abstraction


Algorithmic Paradigm



Language Supported

(Programming Models)



Hardware Architecture




Parallel Programming

(Sequential or parallel) Application Algorithm

Parallel language

and other tools

User (Programmer)

(sequential or parallel) Source program



Compiler (including Preprocessor,

Assembler, and Linker)

Run-Time Support

and Other Libraries

Native Parallel Code

Parallel Platform (OS+Hardware)

Native Programming Model

  • The lowest-level, user-visible programming model provided by a specific parallel computer platform.

  • Example: Power C in SGI Power Challenge, shmem in T3D

  • Other Programming models: data parallel (HPF), message passing (MPI) can be implemented on top of Power C.

Algorithmic Paradigms (Engineering) - Parallel Track

  • Compute-Interact

  • Work-Pool

  • Divide and Conquer

  • Pipelining (Data Stream)

  • Master-Slave

Data vs. Control Parallelism

  • Data parallelism:

    • Multiple, complete functional units apply sameoperation ``simultaneously’’ to different elements of data set.

      • E.g., Divide the domain evenly among the PEs and each PE performs the same task

    • Hardware: SIMD/MIMD machine

    • Data Parallel Programming: HPF (High Performance Fortran), C*

Data vs. Control Parallelism

  • Control parallelism:

    • Apply distinct operations to data elements concurrently.

    • Outputs of operations are fed in as inputs to other operations, in an arbitrary way.

    • The flow of data forms an arbitrary graph.

    • A pipeline is a special case of this, where the graph is just a single path.





Synchronous Interaction




Synchronous Interaction

Work Pool

Get jobs from pool

Divide and Conquer




Merge results

Pipelined (Data Stream)

Edge Detection

Edge Linking

Line Generation

Task 1

Task 2

Task 2

Pipelined computation:

  • Divide computation into a number of stages.

  • Devote separate functional units to each stage;

  • if each completes in the same time, then, once pipe is full, the throughput of the pipeline is 1 result per clock.



(assign jobs)




Algorithmic Paradigms (science) - Sequential Track

  • Divide-and-Conquer

  • Dynamic Programming

  • Branch-and-Bound

  • Backtracking

  • Greedy

    ***** Parallel Versions ??

Parallel Programming Models

Not talking about “language” !!

Programming Models

  • Homogeneity: refers to the similarity of component processes in a parallel program.

Programming Models

  • SPMD: Single-Program-Multiple-Data, programs are homogeneous

  • MPMD: Multiple-Program-Multiple-Data, programs are heterogeneous

  • SIMD: Single-Instruction-Multiple-Data, restricted form of SPMD, all processors execute the same instruction at the same time. Restricted form of SPMD

Programming Models

  • Both SPMD and MPMD are MIMD -- different instructions can be executed by different processes at the same time.

What is SPMD?

  • Single Program, Multiple Data

  • Same program runs everywhere

  • Restriction on the general message-passing model (MPMD)

  • Most venders only support SPMD parallel programs

What is SPMD?

  • General message-passing model (MPMD) can be emulated

  • A data-parallel program refers to an SPMD program in general.

  • Defined by Alan Karp [1987].

MPMD code:

parbegin {





SPMD code:

Main( )


myid = getid ( );

if (myid=0) A;

elseif (myid=1) B;

else (myid=2) C;


An Example of SPMD and MPMD Code



  • A

  • B

  • C

node 0

node 1

node 2



  • Main( )

  • {

  • myid = getid ( );

  • if (myid=0) A;

  • elseif (myid=1) B;

  • else (myid=2) C;

  • }

  • Main( )

  • {

  • myid = getid ( );

  • if (myid=0) A;

  • elseif (myid=1) B;

  • else (myid=2) C;

  • }

  • Main( )

  • {

  • myid = getid ( );

  • if (myid=0) A;

  • elseif (myid=1) B;

  • else (myid=2) C;

  • }

node 0

node 1

node 2

SPMD Programming

  • Two major phases:

    • (1) data distribution choice: determine the mapping of data onto nodes

    • (2) parallel program generation: translate sequential algorithm into the SPMD program (only write one program !!).

Parallel Programming base on SPMD (4 main tasks)

(1) Get node and environmental information:

How many nodes in the system?

Who am I ?

(2) Access data: convert local-to-global and global-to-local indexes

(3) Insert message-passing primitives to exchange data (implicitly or explicitly)

(4) Carry out operations on directly accessible data (local operation, e.g., C code)

Programming Languages

Lots of names !!

Programming Languages

  • Implicit Parallel (KAP)

  • Data Parallel (Fortran 90, HPF, CM Fortran,..)

  • Message-Passing (MPI, PVM, CMMD, NX, Active Message, Fast Message, P4, MPL, LAM, Express)

  • Shared-Variable (X3H5)

  • Hybrid : MPI-HPF, Split-C, MPI-Java (SRG)

Data Parallel Language

main( ) {

double local[N], tmp[N], pi, w;

long i,j, N=100000;


forall (I=0;i<N;I++) {

local[i]= (i-0.5)*w;

tmp[i] = 4.0/(…


pi = sum(tmp);


Data Parallel Model

  • Single threading (from user’s viewpoint): one process + one thread of control. Just like a sequential program.

  • Global naming space: all variable reside in a single address space.

  • Parallel operations on aggregate data structure: sum( ).

Data Parallel Model

  • Loosely synchronization: implicit synchronization after every statement. (Compared with tight synchronization in an SIMD system -- on every instruction)

Message-Passing Language




C Code




Comm Subroutine

Communication Network

Message-Passing Model

  • Multithreading: multiple processes simultaneously executing

  • Separate address space: local variables are not visible to other processes.

Message-Passing Model

  • Explicit allocation: both workload and data are explicitly allocated to the processes by the user.

  • Explicit interactions: communication, synchronization, aggregation,...

  • Asynchronous: the processes execute asynchronously.

Message Passing Programming Model

  • Message-passing programming is more directly painful, but it tends to build locality of reference from the start.

  • The end result seems to be:

    • for scalable, highly parallel, high-performance results, locality of reference will always be central.

double local, pi, w;

long I, taskid;

long numtask;

w= 1.0/N

#pragma shared (pi, w)

#pragma local (I, local)


#pragma pfor iterate (I=0;N;I)

for (I=0; I<N; I++) {

local = (I+0.5) * w;


#pragma critical



Shared variable Model

Shared Variable Model

  • Single address space (similar to data parallel model)

  • Multithreading and asynchronous (similar to message-passing)

  • Communication is done implicitly through shared reads and writes of variables.

  • Synchronization is explicit.

Shared-Memory Programming Model

  • All data shared and visible by executing threads

  • Shared-memory program starts out looking simpler, but memory locality forces one to do some strange transformations

  • Shared-Memory programming standards: ANSI X3H5 (1993), POSIX Threads (Pthreads), OpenMP, SGI Power C


  • Parallel programming has lagged far behind the advances of parallel hardware

  • Compared to the sequential counterparts, today’s parallel system software and application software are few in quantity and primitive in functionality.

  • Likely to continue !!

  • Login