Introduction to message passing
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

Introduction to Message Passing PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on
  • Presentation posted in: General

Introduction to Message Passing. CMSC 34000 Lecture 3 1/11/05. Class goals. Parallel hardware and software issues First look at actual algorithm (trapezoidal rule for numerical integration) Introduction to message passing in MPI Networks and the cost of communication.

Download Presentation

Introduction to Message Passing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Introduction to message passing

Introduction to Message Passing

CMSC 34000

Lecture 3

1/11/05


Class goals

Class goals

  • Parallel hardware and software issues

  • First look at actual algorithm (trapezoidal rule for numerical integration)

  • Introduction to message passing in MPI

  • Networks and the cost of communication


Parallel hardware flynn s taxonomy

Parallel Hardware (Flynn’s Taxonomy)

  • SISD

  • MIMD

  • SIMD

  • MISD

    S = Single, M = Multiple

    I = Instruction stream

    D = Data stream


Von neumann modern

CPU (control and arithmetic)

Memory (main & register

Data/instruction transfer bottleneck

Pipelining (multiple instructions operating simultaneously)

Vectorizing (single instruction acts on vector register

Cache -- memory hierarchy

Von Neumann & Modern


Simd mimd

Single CPU for control

Many (scalar) ALUs with registers

One clock

Many CPUs

Each has control and ALU

Memory may be “shared” or “distributed”

Synchronized?

SIMD / MIMD


Shared memory mimd

Shared memory MIMD

  • Bus (contention)

  • Switch (expensive)

  • Cache-coherency?


Distributed memory mimd

Distributed memory MIMD

  • General interconnection network (e.g. CS Linux system)

  • Each processor has its own memory

  • To share information, processors must pass (and receive) messages that go over the network.

  • Topology is very important


Different mesh topologies

Different mesh topologies

  • Totally connected

  • Linear array/ring

  • Hypercube

  • Mesh/Torus

  • Tree / hypertree

  • Ethernet…

  • And others


Issues to consider

Issues to consider

  • Routing (shortest path = best-case cost of single message)

  • Contention - multiple messages between different processors must share a wire

  • Programming: would like libraries that hide all this (somehow)


Numerical integration

Numerical integration

  • Approximate

  • Using quadrature:

  • Repeated subdivision:


Finite differences

Finite differences

  • On (0,1):

  • At endpoints


System of equations

System of Equations

  • Algebraic system of equations at each point

  • “Nearest neighbor stencil”

  • A row of the matrix looks like


Parallel strategy integration

Parallel strategy: Integration

  • Divide [a,b] into p intervals

  • Approximation on each subinterval

  • Sum approximations over each processor

    • How do we communicate?

    • Broadcast / reduction


Parallel strategy finite differences

Parallel strategy: Finite differences

  • How do we multiply the matrix by a vector (needed in Krylov subspace methods)?

  • Each processor owns:

    • A range of points

    • A range of matrix rows

    • A range of vector entries

  • To multiply by a vector (linear array)

    • Share the values at endpoints with neighbors


Spmd integration

SPMD (Integration)

  • Single program running on multiple data

    • Summation over intervals

    • Particular points are different

  • Instances of program can talk to each other

    • All must share information at the same time

    • Synchronization


Introduction to message passing

MPI

  • Message Passing Interface

  • Developed in 1990’s

  • Standard for

    • Sharing message

    • Collective communication

    • Logical topologies

    • Etc


Integration in mpi

Integration in MPI

  • Python bindings developed by Pat Miller (LLNL)

  • Ignore data types, memory size for now

  • Look at sample code


Two versions

explicit send and receive

O(p) communication cost (at best)

Broadcast sends to all processes

Reduce collects information to a single process

Run-time depends on topology, implementation

Two versions


Fundamental model of a message

Fundamental model of a message

  • Processor p “sends”

  • Processor q “receives”

  • Information needed:

    • Address (to read from/write to)

    • Amount of data being sent

    • Type of data

    • Tag to screen the messages

    • How much data actually received?


Mpi fundamentals

MPI Fundamentals

  • MPI_COMM_WORLD

  • MPI_Init() // import mpi

  • MPI_Comm_size() // mpi.size

  • MPI_Comm_rank() // mpi.rank

  • MPI_Finalize() // N/A


Mpi fundamentals1

send

receive

non-blocking versions

broadcast

reduce

other collective ops

MPI Fundamentals


Communication costs over a network

Communication costs over a network

  • Send, broadcast, reduce

    • Linear array

    • Point-to-point

    • Binary tree


Getting started with mpi on cs machines

Getting started with MPI on CS machines

  • Machines available (Debian unstable)

    • bombadil, clark, guts, garfield

  • mpich (installed on our system already)

    • mpicc is the compiler (mpic++, mpif77,etc)

    • mpirun -np x -machinefile hostsexecutableargs)

  • download pyMPI & build in home directory

    • http://sourceforge.net/projects/pympi/

    • ./configure --prefix = /home/<you>

    • builds out of box (fingers crossed)


  • Login