introduction to message passing
Skip this Video
Download Presentation
Introduction to Message Passing

Loading in 2 Seconds...

play fullscreen
1 / 23

Introduction to Message Passing - PowerPoint PPT Presentation

  • Uploaded on

Introduction to Message Passing. CMSC 34000 Lecture 3 1/11/05. Class goals. Parallel hardware and software issues First look at actual algorithm (trapezoidal rule for numerical integration) Introduction to message passing in MPI Networks and the cost of communication.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Introduction to Message Passing' - desiree-bowers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
introduction to message passing

Introduction to Message Passing

CMSC 34000

Lecture 3


class goals
Class goals
  • Parallel hardware and software issues
  • First look at actual algorithm (trapezoidal rule for numerical integration)
  • Introduction to message passing in MPI
  • Networks and the cost of communication
parallel hardware flynn s taxonomy
Parallel Hardware (Flynn’s Taxonomy)
  • SISD
  • MIMD
  • SIMD
  • MISD

S = Single, M = Multiple

I = Instruction stream

D = Data stream

von neumann modern
CPU (control and arithmetic)

Memory (main & register

Data/instruction transfer bottleneck

Pipelining (multiple instructions operating simultaneously)

Vectorizing (single instruction acts on vector register

Cache -- memory hierarchy

Von Neumann & Modern
simd mimd
Single CPU for control

Many (scalar) ALUs with registers

One clock

Many CPUs

Each has control and ALU

Memory may be “shared” or “distributed”


shared memory mimd
Shared memory MIMD
  • Bus (contention)
  • Switch (expensive)
  • Cache-coherency?
distributed memory mimd
Distributed memory MIMD
  • General interconnection network (e.g. CS Linux system)
  • Each processor has its own memory
  • To share information, processors must pass (and receive) messages that go over the network.
  • Topology is very important
different mesh topologies
Different mesh topologies
  • Totally connected
  • Linear array/ring
  • Hypercube
  • Mesh/Torus
  • Tree / hypertree
  • Ethernet…
  • And others
issues to consider
Issues to consider
  • Routing (shortest path = best-case cost of single message)
  • Contention - multiple messages between different processors must share a wire
  • Programming: would like libraries that hide all this (somehow)
numerical integration
Numerical integration
  • Approximate
  • Using quadrature:
  • Repeated subdivision:
finite differences
Finite differences
  • On (0,1):
  • At endpoints
system of equations
System of Equations
  • Algebraic system of equations at each point
  • “Nearest neighbor stencil”
  • A row of the matrix looks like
parallel strategy integration
Parallel strategy: Integration
  • Divide [a,b] into p intervals
  • Approximation on each subinterval
  • Sum approximations over each processor
    • How do we communicate?
    • Broadcast / reduction
parallel strategy finite differences
Parallel strategy: Finite differences
  • How do we multiply the matrix by a vector (needed in Krylov subspace methods)?
  • Each processor owns:
    • A range of points
    • A range of matrix rows
    • A range of vector entries
  • To multiply by a vector (linear array)
    • Share the values at endpoints with neighbors
spmd integration
SPMD (Integration)
  • Single program running on multiple data
    • Summation over intervals
    • Particular points are different
  • Instances of program can talk to each other
    • All must share information at the same time
    • Synchronization
  • Message Passing Interface
  • Developed in 1990’s
  • Standard for
    • Sharing message
    • Collective communication
    • Logical topologies
    • Etc
integration in mpi
Integration in MPI
  • Python bindings developed by Pat Miller (LLNL)
  • Ignore data types, memory size for now
  • Look at sample code
two versions
explicit send and receive

O(p) communication cost (at best)

Broadcast sends to all processes

Reduce collects information to a single process

Run-time depends on topology, implementation

Two versions
fundamental model of a message
Fundamental model of a message
  • Processor p “sends”
  • Processor q “receives”
  • Information needed:
    • Address (to read from/write to)
    • Amount of data being sent
    • Type of data
    • Tag to screen the messages
    • How much data actually received?
mpi fundamentals
MPI Fundamentals
  • MPI_Init() // import mpi
  • MPI_Comm_size() // mpi.size
  • MPI_Comm_rank() // mpi.rank
  • MPI_Finalize() // N/A
mpi fundamentals1


non-blocking versions



other collective ops

MPI Fundamentals
communication costs over a network
Communication costs over a network
  • Send, broadcast, reduce
    • Linear array
    • Point-to-point
    • Binary tree
getting started with mpi on cs machines
Getting started with MPI on CS machines
  • Machines available (Debian unstable)
    • bombadil, clark, guts, garfield
  • mpich (installed on our system already)
    • mpicc is the compiler (mpic++, mpif77,etc)
    • mpirun -np x -machinefile hostsexecutableargs)
  • download pyMPI & build in home directory
    • ./configure --prefix = /home/
    • builds out of box (fingers crossed)