Introduction to Message Passing - PowerPoint PPT Presentation

Introduction to message passing
Download
1 / 23

  • 116 Views
  • Uploaded on
  • Presentation posted in: General

Introduction to Message Passing. CMSC 34000 Lecture 3 1/11/05. Class goals. Parallel hardware and software issues First look at actual algorithm (trapezoidal rule for numerical integration) Introduction to message passing in MPI Networks and the cost of communication.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Introduction to Message Passing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Introduction to message passing

Introduction to Message Passing

CMSC 34000

Lecture 3

1/11/05


Class goals

Class goals

  • Parallel hardware and software issues

  • First look at actual algorithm (trapezoidal rule for numerical integration)

  • Introduction to message passing in MPI

  • Networks and the cost of communication


Parallel hardware flynn s taxonomy

Parallel Hardware (Flynn’s Taxonomy)

  • SISD

  • MIMD

  • SIMD

  • MISD

    S = Single, M = Multiple

    I = Instruction stream

    D = Data stream


Von neumann modern

CPU (control and arithmetic)

Memory (main & register

Data/instruction transfer bottleneck

Pipelining (multiple instructions operating simultaneously)

Vectorizing (single instruction acts on vector register

Cache -- memory hierarchy

Von Neumann & Modern


Simd mimd

Single CPU for control

Many (scalar) ALUs with registers

One clock

Many CPUs

Each has control and ALU

Memory may be “shared” or “distributed”

Synchronized?

SIMD / MIMD


Shared memory mimd

Shared memory MIMD

  • Bus (contention)

  • Switch (expensive)

  • Cache-coherency?


Distributed memory mimd

Distributed memory MIMD

  • General interconnection network (e.g. CS Linux system)

  • Each processor has its own memory

  • To share information, processors must pass (and receive) messages that go over the network.

  • Topology is very important


Different mesh topologies

Different mesh topologies

  • Totally connected

  • Linear array/ring

  • Hypercube

  • Mesh/Torus

  • Tree / hypertree

  • Ethernet…

  • And others


Issues to consider

Issues to consider

  • Routing (shortest path = best-case cost of single message)

  • Contention - multiple messages between different processors must share a wire

  • Programming: would like libraries that hide all this (somehow)


Numerical integration

Numerical integration

  • Approximate

  • Using quadrature:

  • Repeated subdivision:


Finite differences

Finite differences

  • On (0,1):

  • At endpoints


System of equations

System of Equations

  • Algebraic system of equations at each point

  • “Nearest neighbor stencil”

  • A row of the matrix looks like


Parallel strategy integration

Parallel strategy: Integration

  • Divide [a,b] into p intervals

  • Approximation on each subinterval

  • Sum approximations over each processor

    • How do we communicate?

    • Broadcast / reduction


Parallel strategy finite differences

Parallel strategy: Finite differences

  • How do we multiply the matrix by a vector (needed in Krylov subspace methods)?

  • Each processor owns:

    • A range of points

    • A range of matrix rows

    • A range of vector entries

  • To multiply by a vector (linear array)

    • Share the values at endpoints with neighbors


Spmd integration

SPMD (Integration)

  • Single program running on multiple data

    • Summation over intervals

    • Particular points are different

  • Instances of program can talk to each other

    • All must share information at the same time

    • Synchronization


Introduction to message passing

MPI

  • Message Passing Interface

  • Developed in 1990’s

  • Standard for

    • Sharing message

    • Collective communication

    • Logical topologies

    • Etc


Integration in mpi

Integration in MPI

  • Python bindings developed by Pat Miller (LLNL)

  • Ignore data types, memory size for now

  • Look at sample code


Two versions

explicit send and receive

O(p) communication cost (at best)

Broadcast sends to all processes

Reduce collects information to a single process

Run-time depends on topology, implementation

Two versions


Fundamental model of a message

Fundamental model of a message

  • Processor p “sends”

  • Processor q “receives”

  • Information needed:

    • Address (to read from/write to)

    • Amount of data being sent

    • Type of data

    • Tag to screen the messages

    • How much data actually received?


Mpi fundamentals

MPI Fundamentals

  • MPI_COMM_WORLD

  • MPI_Init() // import mpi

  • MPI_Comm_size() // mpi.size

  • MPI_Comm_rank() // mpi.rank

  • MPI_Finalize() // N/A


Mpi fundamentals1

send

receive

non-blocking versions

broadcast

reduce

other collective ops

MPI Fundamentals


Communication costs over a network

Communication costs over a network

  • Send, broadcast, reduce

    • Linear array

    • Point-to-point

    • Binary tree


Getting started with mpi on cs machines

Getting started with MPI on CS machines

  • Machines available (Debian unstable)

    • bombadil, clark, guts, garfield

  • mpich (installed on our system already)

    • mpicc is the compiler (mpic++, mpif77,etc)

    • mpirun -np x -machinefile hostsexecutableargs)

  • download pyMPI & build in home directory

    • http://sourceforge.net/projects/pympi/

    • ./configure --prefix = /home/<you>

    • builds out of box (fingers crossed)


  • Login