Introduction to Message Passing - PowerPoint PPT Presentation

Introduction to message passing
1 / 23

  • Uploaded on
  • Presentation posted in: General

Introduction to Message Passing. CMSC 34000 Lecture 3 1/11/05. Class goals. Parallel hardware and software issues First look at actual algorithm (trapezoidal rule for numerical integration) Introduction to message passing in MPI Networks and the cost of communication.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Introduction to Message Passing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Introduction to message passing

Introduction to Message Passing

CMSC 34000

Lecture 3


Class goals

Class goals

  • Parallel hardware and software issues

  • First look at actual algorithm (trapezoidal rule for numerical integration)

  • Introduction to message passing in MPI

  • Networks and the cost of communication

Parallel hardware flynn s taxonomy

Parallel Hardware (Flynn’s Taxonomy)

  • SISD

  • MIMD

  • SIMD

  • MISD

    S = Single, M = Multiple

    I = Instruction stream

    D = Data stream

Von neumann modern

CPU (control and arithmetic)

Memory (main & register

Data/instruction transfer bottleneck

Pipelining (multiple instructions operating simultaneously)

Vectorizing (single instruction acts on vector register

Cache -- memory hierarchy

Von Neumann & Modern

Simd mimd

Single CPU for control

Many (scalar) ALUs with registers

One clock

Many CPUs

Each has control and ALU

Memory may be “shared” or “distributed”



Shared memory mimd

Shared memory MIMD

  • Bus (contention)

  • Switch (expensive)

  • Cache-coherency?

Distributed memory mimd

Distributed memory MIMD

  • General interconnection network (e.g. CS Linux system)

  • Each processor has its own memory

  • To share information, processors must pass (and receive) messages that go over the network.

  • Topology is very important

Different mesh topologies

Different mesh topologies

  • Totally connected

  • Linear array/ring

  • Hypercube

  • Mesh/Torus

  • Tree / hypertree

  • Ethernet…

  • And others

Issues to consider

Issues to consider

  • Routing (shortest path = best-case cost of single message)

  • Contention - multiple messages between different processors must share a wire

  • Programming: would like libraries that hide all this (somehow)

Numerical integration

Numerical integration

  • Approximate

  • Using quadrature:

  • Repeated subdivision:

Finite differences

Finite differences

  • On (0,1):

  • At endpoints

System of equations

System of Equations

  • Algebraic system of equations at each point

  • “Nearest neighbor stencil”

  • A row of the matrix looks like

Parallel strategy integration

Parallel strategy: Integration

  • Divide [a,b] into p intervals

  • Approximation on each subinterval

  • Sum approximations over each processor

    • How do we communicate?

    • Broadcast / reduction

Parallel strategy finite differences

Parallel strategy: Finite differences

  • How do we multiply the matrix by a vector (needed in Krylov subspace methods)?

  • Each processor owns:

    • A range of points

    • A range of matrix rows

    • A range of vector entries

  • To multiply by a vector (linear array)

    • Share the values at endpoints with neighbors

Spmd integration

SPMD (Integration)

  • Single program running on multiple data

    • Summation over intervals

    • Particular points are different

  • Instances of program can talk to each other

    • All must share information at the same time

    • Synchronization

Introduction to message passing


  • Message Passing Interface

  • Developed in 1990’s

  • Standard for

    • Sharing message

    • Collective communication

    • Logical topologies

    • Etc

Integration in mpi

Integration in MPI

  • Python bindings developed by Pat Miller (LLNL)

  • Ignore data types, memory size for now

  • Look at sample code

Two versions

explicit send and receive

O(p) communication cost (at best)

Broadcast sends to all processes

Reduce collects information to a single process

Run-time depends on topology, implementation

Two versions

Fundamental model of a message

Fundamental model of a message

  • Processor p “sends”

  • Processor q “receives”

  • Information needed:

    • Address (to read from/write to)

    • Amount of data being sent

    • Type of data

    • Tag to screen the messages

    • How much data actually received?

Mpi fundamentals

MPI Fundamentals


  • MPI_Init() // import mpi

  • MPI_Comm_size() // mpi.size

  • MPI_Comm_rank() // mpi.rank

  • MPI_Finalize() // N/A

Mpi fundamentals1



non-blocking versions



other collective ops

MPI Fundamentals

Communication costs over a network

Communication costs over a network

  • Send, broadcast, reduce

    • Linear array

    • Point-to-point

    • Binary tree

Getting started with mpi on cs machines

Getting started with MPI on CS machines

  • Machines available (Debian unstable)

    • bombadil, clark, guts, garfield

  • mpich (installed on our system already)

    • mpicc is the compiler (mpic++, mpif77,etc)

    • mpirun -np x -machinefile hostsexecutableargs)

  • download pyMPI & build in home directory


    • ./configure --prefix = /home/<you>

    • builds out of box (fingers crossed)

  • Login