cs668 lecture 9 objectives n.
Skip this Video
Download Presentation
CS668 Lecture 9 Objectives

Loading in 2 Seconds...

play fullscreen
1 / 40

CS668 Lecture 9 Objectives - PowerPoint PPT Presentation

  • Uploaded on

CS668 Lecture 9 Objectives. Material from Chapter 9 Show how to implement manager-worker programs Parallel Algorithms for Document Classification Parallel Algorithms for Clustering. Outline. Creating communicators Non-blocking communications Implementation Pipelining Tasks

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'CS668 Lecture 9 Objectives' - wilson

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
cs668 lecture 9 objectives
CS668 Lecture 9 Objectives
  • Material from Chapter 9
  • Show how to implement manager-worker programs
  • Parallel Algorithms for Document Classification
  • Parallel Algorithms for Clustering
  • Creating communicators
  • Non-blocking communications
  • Implementation
  • Pipelining Tasks
  • Clustering Vectors
implementation of a very simple document classifier
Implementation of a Very Simple Document Classifier
  • Manager/Worker Design Strategy
  • Manager description
    • create initial tasks
    • communicate to/from workers
  • Worker description
    • receive tasks
    • Alternate between task-computation and communication with master
structure of main program manager worker paradigm
Structure of Main program:Manager/Worker Paradigm

MPI_Init (&argc, &argv);

// what is my rank?

MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

// how many processors are there?

MPI_Comm_size(MPI_COMM_WORLD, &p);

if (myid == 0)



Worker(myid, p);




more mpi functions
More MPI functions
  • MPI_Abort
  • MPI_Comm_split
  • MPI_Isend, MPI_Irecv, MPI_Wait
  • MPI_Probe
  • MPI_Get_count
  • MPI_Testsome
mpi abort
  • A “quick and dirty” way for one process to terminate all processes in a specified communicator
  • Example use: If manager cannot allocate memory needed to store document profile vectors

int MPI_Abort (

MPI_Comm comm, /* Communicator */

int error_code)/* Value returned to

calling environment */

creating a workers only communicator
Creating a Workers-only Communicator
  • To support workers-only broadcast, need workers-only communicator
  • Can use MPI_Comm_split
  • Excluded processes (e.g., Manager) passes MPI_UNDEFINED as the value of split_key, meaning it will not be part of any new communicator
workers only communicator
Workers-only Communicator

int id;

MPI_Comm worker_comm;


if (!id) /* Manager */


MPI_UNDEFINED, id, &worker_comm);

else /* Worker */

MPI_Comm_split (MPI_COMM_WORLD, 0,

id, &worker_comm);

nonblocking send receive
Nonblocking Send / Receive
  • Persistent communications reduce overhead when repeated calls to same point-to-point message passing routines
  • MPI_Isend, MPI_Irecv initiate operation
  • MPI_Wait blocks until operation complete
  • Calls can be made early
    • MPI_Isend as soon as value(s) assigned
    • MPI_Irecv as soon as buffer available
  • Can eliminate a message copying step
  • Allows communication / computation overlap
function mpi isend

Pointer to object that identifies

communication operation

Function MPI_Isend

int MPI_Isend (

void *buffer,

int cnt,

MPI_Datatype dtype,

int dest,

int tag,

MPI_Comm comm,

MPI_Request *handle


function mpi irecv

Pointer to object that identifies

communication operation

Function MPI_Irecv

int MPI_Irecv (

void *buffer,

int cnt,

MPI_Datatype dtype,

int src,

int tag,

MPI_Comm comm,

MPI_Request *handle


function mpi wait
Function MPI_Wait

int MPI_Wait (

MPI_Request *handle,

MPI_Status *status


Blocks until operation associated with pointer handle completes.

status points to object containing info on received message

receiving problem
Receiving Problem
  • Worker does not know length of message it will receive
  • Example, the length of File Path Name
  • Alternatives
    • Allocate huge buffer
    • Check length of incoming message, then allocate buffer
  • We’ll take the second alternative
function mpi probe
Function MPI_Probe

int MPI_Probe (

int src,

int tag,

MPI_Comm comm,

MPI_Status *status


Works for any Send.

Blocks until message is available to be received from

process with rank src with message tag tag; status pointer gives info

on message size.

function mpi get count
Function MPI_Get_count

int MPI_Get_count (

MPI_Status *status,

MPI_Datatype dtype,

int *cnt


cnt returns the number of elements in message

mpi testsome
  • Non-blocking test
  • Often need to check whether one or more messages have arrived
  • Manager posts a nonblocking receive to each worker process
  • Builds an array of handles or request objects
  • Testsome allows manager to determine how many messages have arrived
function mpi testsome
Function MPI_Testsome

int MPI_Testsome (

int in_cnt, /* IN - Number of

nonblocking receives to check */

MPI_Request *handlearray, /* IN -

Handles of pending receives */

int *out_cnt, /* OUT - Number of

completed communications */

int *index_array, /* OUT - Indices of

completed communications */

MPI_Status *status_array) /* OUT -

Status records for completed comms */

document classification problem
Document Classification Problem
  • Search directories, subdirectories for documents (look for .html, .txt, .tex, etc.)
  • Using a dictionary of key words, create a profile vector for each document
  • Store profile vectors
partitioning and communication
Partitioning and Communication
  • Most time spent reading documents and generating profile vectors
  • Create two primitive tasks for each document
agglomeration and mapping
Agglomeration and Mapping
  • Number of tasks not known at compile time
  • Tasks do not communicate with each other
  • Time needed to perform tasks varies widely
  • Strategy: map tasks to processes at run time
manager worker style algorithm
Manager/worker-style Algorithm
  • Task/Functional Partitioning
  • Domain/Data Partitioning
manager pseudocode
Manager Pseudocode

Identify documents

Receive dictionary size from worker 0

Allocate matrix to store document vectors


Receive message from worker

if message contains document vector

Store document vector


if documents remain then Send worker file name

else Send worker termination message


until all workers terminated

Write document vectors to file

worker pseudocode
Worker Pseudocode

Send first request for work to manager

if worker 0 then

Read dictionary from file


Broadcast dictionary among workers

Build hash table from dictionary

if worker 0 then

Send dictionary size to manager



Receive file name from manager

if file name is NULL then terminate endif

Read document, generate document vector

Send document vector to manager


  • Finding middle ground between pre-allocation and one-at-a-time allocation of file paths
  • Pipelining of document processing
allocation alternatives

Load imbalance


Allocation Alternatives






Documents Allocated per Request

pipelined manager pseudocode
Pipelined Manager Pseudocode

a 0 {assigned jobs}

j  0 {available jobs}

w  0 {workers waiting for assignment}


if (j > 0) and (w > 0) then

assign job to worker

j  j– 1; w  w– 1; a  a + 1

elseif (j > 0) then

handle an incoming message from workers

increment w


get another job

increment j


until (a = n) and (w = p)

  • Manager/worker paradigm
    • Dynamic number of tasks
    • Variable task lengths
    • No communications between tasks
  • New tools for “kit”
    • Create manager/worker program
    • Create workers-only communicator
    • Non-blocking send/receive
    • Testing for completed communications
  • Next Step: Cluster Profile Vectors
k means clustering
K-Means Clustering
  • Assumes documents are real-valued vectors.
  • Assumes distance function on vector pairs
  • Clusters based on centroids (aka the center of gravity or mean) of points in a cluster, c:
  • Reassignment of instances to clusters is based on distance of vector to the current cluster centroids.
    • (Or one can equivalently phrase it in terms of similarities)
k means algorithm
K-Means Algorithm

Let d be the distance measure between instances.

Select k random instances {s1, s2,… sk} as seeds.

Until clustering converges or other stopping criterion:

For each instance xi:

Assign xi to the cluster cjsuch that d(xi, sj) is minimal.

// Now Update the seeds to the centroid of each cluster)

For each cluster cj

sj = (cj)

k means example k 2

Pick seeds

Reassign clusters

Compute centroids

Reassign clusters




Compute centroids




K Means Example(K=2)

Reassign clusters


termination conditions
Termination conditions
  • Desire that docs in a cluster are unchanged
  • Several possibilities, e.g.,
    • A fixed number of iterations.
    • Doc partition unchanged.
    • Centroid positions don’t change.
    • We’ll choose termination when only small fraction change (threshold value)
    • Describe Manager/Worker pseudo-code that implements the K-means algorithm in parallel
    • What data partitioning for parallelism?
    • How are cluster centers updated and distributed?
  • objects to be clustered are evenly partitioned among all processes
  • cluster centers are replicated
  • Global-sum reduction on cluster centers is performed at the end of each iteration to generate the new cluster centers.
  • Use MPI_Bcast and MPI_Allreduce