introduction l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Introduction PowerPoint Presentation
Download Presentation
Introduction

Loading in 2 Seconds...

play fullscreen
1 / 60

Introduction - PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on

Introduction. Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified by Rajeev Alur for CIS 640 at Penn, Spring 2009. Moore’s Law. Transistor count still rising. Clock speed flattening sharply. Still on some of your desktops: The Uniprocesor.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Introduction' - hao


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
introduction

Introduction

Companion slides for

The Art of Multiprocessor Programming

by Maurice Herlihy & Nir Shavit

Modified by Rajeev Alur

for CIS 640 at Penn, Spring 2009

moore s law
Moore’s Law

Transistor count still rising

Clock speed flattening sharply

Art of Multiprocessor Programming

still on some of your desktops the uniprocesor
Still on some of your desktops: The Uniprocesor

cpu

memory

Art of Multiprocessor Programming

in the enterprise the shared memory multiprocessor smp
In the Enterprise: The Shared Memory Multiprocessor(SMP)

cache

cache

cache

Bus

Bus

shared memory

Art of Multiprocessor Programming

your new desktop the multicore processor cmp
Your New Desktop: The Multicore Processor(CMP)

Sun T2000

Niagara

All on the

same chip

cache

cache

cache

Bus

Bus

shared memory

Art of Multiprocessor Programming

multicores are here
Multicores Are Here

“Intel's Intel ups ante with 4-core chip. New microprocessor, due this year, will be faster, use less electricity...” [San Fran Chronicle]

“AMD will launch a dual-core version of its Opteron server processor at an event in New York on April 21.” [PC World]

“Sun’s Niagara…will have eight cores, each core capable of running 4 threads in parallel, for 32 concurrently running threads. ….” [The Inquirer]

Art of Multiprocessor Programming

why do we care
Why do we care?

Time no longer cures software bloat

The “free ride” is over

When you double your program’s path length

You can’t just wait 6 months

Your software must somehow exploit twice as much concurrency

Art of Multiprocessor Programming

traditional scaling process
Traditional Scaling Process

7x

Speedup

3.6x

1.8x

User code

Traditional

Uniprocessor

Time: Moore’s law

Art of Multiprocessor Programming

multicore scaling process
Multicore Scaling Process

7x

3.6x

Speedup

1.8x

User code

Multicore

Unfortunately, not so simple…

Art of Multiprocessor Programming

real world scaling process
Real-World Scaling Process

Speedup

2.9x

2x

1.8x

User code

Multicore

Parallelization and Synchronization

require great care…

Art of Multiprocessor Programming

multicore programming course overview
Multicore Programming: Course Overview

Fundamentals

Models, algorithms, impossibility

Real-World programming

Architectures

Techniques

Topics not in Textbook

Memory models and system-level concurrency libraries

High-level programming abstractions

Art of Multiprocessor Programming

a zoo of terms
A Zoo of Terms
  • Concurrent
  • Parallel
  • Distributed
  • Multicore

What do they all mean? How do they differ?

concurrent computing
Concurrent Computing
  • Programs designed as a collection of interacting threads/processes
    • Logical/programming abstraction
    • May be implemented on single processor by interleaving or on multiple processors or on distributed computers
    • Coordination/synchronization mechanism in a model of concurrency may be realized in many ways in an implementation
parallel computing
Parallel Computing
  • Computations that execute simultaneously to solve a common problem (more efficiently)
    • Parallel algorithms: Which problems can have speed-up given multiple execution units?
    • Parallelism can be at many levels (e.g. bit-level, instruction-level, data path)
    • Grid computing: Branch of parallel computing where problems are solved on clusters of computers (interacting by message passing)
    • Multicore computing: Branch of parallel computing focusing on multiple execution units on same chip (interacting by shared memory)
distributed computing
Distributed Computing
  • Involves multiple agents/programs (possibly with different computational tasks) with multiple computational resources (computers, multiprocessors, network)
    • Many examples of contemporary software (e.g. web services) are distributed systems
    • Heterogeneous nature, and range of time scales (web access vs local access), make design/programming more challenging
sequential computation
Sequential Computation

thread

memory

object

object

Art of Multiprocessor Programming

concurrent computation
Concurrent Computation

threads

memory

object

object

Art of Multiprocessor Programming

asynchrony
Asynchrony

Sudden unpredictable delays

Cache misses (short)

Page faults (long)

Scheduling quantum used up (really long)

Art of Multiprocessor Programming

model summary
Model Summary

Multiple threads

Sometimes called processes

Single shared memory

Objects live in memory

Unpredictable asynchronous delays

Art of Multiprocessor Programming

road map
Road Map

Textbook focuses on principles first, then practice

Start with idealized models

Look at simplistic problems

Emphasize correctness over pragmatism

“Correctness may be theoretical, but incorrectness has practical impact”

In course, interleaving of chapters from the two parts

Art of Multiprocessor Programming

concurrency jargon
Concurrency Jargon

Hardware

Processors

Software

Threads, processes

Sometimes OK to confuse them, sometimes not.

Art of Multiprocessor Programming

parallel primality testing
Parallel Primality Testing

Challenge

Print primes from 1 to 1010

Given

Ten-processor multiprocessor

One thread per processor

Goal

Get ten-fold speedup (or close)

Art of Multiprocessor Programming

load balancing
Load Balancing

Split the work evenly

Each thread tests range of 109

1

1010

109

2·109

P0

P1

P9

Art of Multiprocessor Programming

procedure for thread i
Procedure for Thread i

void primePrint {

int i = ThreadID.get(); // IDs in {0..9}

for(j = i*109+1, j<(i+1)*109; j++) {

if(isPrime(j))

print(j);

}

}

Art of Multiprocessor Programming

issues
Issues

Higher ranges have fewer primes

Yet larger numbers harder to test

Thread workloads

Uneven

Hard to predict

Art of Multiprocessor Programming

issues26
Issues

Higher ranges have fewer primes

Yet larger numbers harder to test

Thread workloads

Uneven

Hard to predict

Need dynamic load balancing

rejected

Art of Multiprocessor Programming

shared counter
Shared Counter

19

each thread takes a number

18

17

Art of Multiprocessor Programming

procedure for thread i28
Procedure for Thread i

int counter = new Counter(1);

void primePrint {

long j = 0;

while (j < 1010) {

j = counter.getAndIncrement();

if (isPrime(j))

print(j);

}

}

Art of Multiprocessor Programming

procedure for thread i29
Procedure for Thread i

Counter counter = new Counter(1);

void primePrint {

long j = 0;

while (j < 1010) {

j = counter.getAndIncrement();

if (isPrime(j))

print(j);

}

}

Shared counter

object

Art of Multiprocessor Programming

where things reside
Where Things Reside

cache

cache

cache

Bus

Bus

void primePrint {

int i = ThreadID.get(); // IDs in {0..9}

for(j = i*109+1, j<(i+1)*109; j++) {

if(isPrime(j))

print(j);

}

}

Local

variables

code

shared

memory

1

shared counter

Art of Multiprocessor Programming

procedure for thread i31
Procedure for Thread i

Counter counter = new Counter(1);

void primePrint {

long j = 0;

while (j < 1010) {

j = counter.getAndIncrement();

if (isPrime(j))

print(j);

}

}

Stop when every value taken

Art of Multiprocessor Programming

procedure for thread i32
Procedure for Thread i

Counter counter = new Counter(1);

void primePrint {

long j = 0;

while (j < 1010) {

j =counter.getAndIncrement();

if (isPrime(j))

print(j);

}

}

Increment & return each new value

Art of Multiprocessor Programming

counter implementation
Counter Implementation

public class Counter{

private long value;

public long getAndIncrement() {

return value++;

}

}

Art of Multiprocessor Programming

counter implementation34
Counter Implementation

public class Counter {

private long value;

public long getAndIncrement() {

return value++;

}

}

OK for single thread,

not for concurrent threads

Art of Multiprocessor Programming

what it means
What It Means

public class Counter {

private long value;

public long getAndIncrement() {

return value++;

}

}

Art of Multiprocessor Programming

what it means36
What It Means

public class Counter {

private long value;

public long getAndIncrement() {

return value++;

}

}

temp = value;

value = temp + 1;

return temp;

Art of Multiprocessor Programming

not so good

time

Not so good…

Value… 1

2

3

2

read

1

write

2

read

2

write

3

read

1

write

2

Art of Multiprocessor Programming

is this problem inherent
Is this problem inherent?

write

read

read

write

If we could only glue reads and writes…

Art of Multiprocessor Programming

challenge
Challenge

public class Counter {

private long value;

public long getAndIncrement() {

temp = value;

value = temp + 1;

return temp;

}

}

Art of Multiprocessor Programming

challenge40
Challenge

public class Counter {

private long value;

public long getAndIncrement() {

temp = value;

value = temp + 1;

return temp;

}

}

Make these steps atomic (indivisible)

Art of Multiprocessor Programming

hardware solution
Hardware Solution

public class Counter {

private long value;

public long getAndIncrement() {

temp = value;

value = temp + 1;

return temp;

}

}

ReadModifyWrite()

instruction

Art of Multiprocessor Programming

an aside java
An Aside: Java™

public class Counter {

private long value;

public long getAndIncrement() {

synchronized{

temp = value;

value = temp + 1;

}

return temp;

}

}

Art of Multiprocessor Programming

an aside java43
An Aside: Java™

public class Counter {

private long value;

public long getAndIncrement() {

synchronized{

temp = value;

value = temp + 1;

}

return temp;

}

}

Synchronized block

Art of Multiprocessor Programming

an aside java44
An Aside: Java™

public class Counter {

private long value;

public long getAndIncrement() {

synchronized {

temp = value;

value = temp + 1;

}

return temp;

}

}

Mutual Exclusion

Art of Multiprocessor Programming

why do we care45
Why do we care?

We want as much of the code as possible to execute concurrently (in parallel)

A larger sequential part implies reduced performance

Amdahl’s law: this relation is not linear…

Art of Multiprocessor Programming

amdahl s law
Amdahl’s Law

Speedup=

…of computation given nCPUs instead of 1

Art of Multiprocessor Programming

amdahl s law47
Amdahl’s Law

Speedup=

Art of Multiprocessor Programming

amdahl s law48
Amdahl’s Law

Parallel fraction

Speedup=

Art of Multiprocessor Programming

amdahl s law49
Amdahl’s Law

Sequential fraction

Parallel fraction

Speedup=

Art of Multiprocessor Programming

amdahl s law50
Amdahl’s Law

Sequential fraction

Parallel fraction

Speedup=

Number of processors

Art of Multiprocessor Programming

example
Example
  • Ten processors
  • 60% concurrent, 40% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example52

Speedup=2.17=

Example
  • Ten processors
  • 60% concurrent, 40% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example53
Example
  • Ten processors
  • 80% concurrent, 20% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example54

Speedup=3.57=

Example
  • Ten processors
  • 80% concurrent, 20% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example55
Example
  • Ten processors
  • 90% concurrent, 10% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example56

Speedup=5.26=

Example
  • Ten processors
  • 90% concurrent, 10% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example57
Example
  • Ten processors
  • 99% concurrent, 01% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

example58

Speedup=9.17=

Example
  • Ten processors
  • 99% concurrent, 01% sequential
  • How close to 10-fold speedup?

Art of Multiprocessor Programming

the moral
The Moral

Making good use of our multiple processors (cores) means

Finding ways to effectively parallelize our code

Minimize sequential parts

Reduce idle time in which threads wait

Art of Multiprocessor Programming

multicore programming
Multicore Programming

This is what this course is about…

The % that is not easy to make concurrent yet may have a large impact on overall speedup

Art of Multiprocessor Programming