1 / 90

CS 267 Dense Linear Algebra: History and Structure, Parallel Matrix Multiplication

CS 267 Dense Linear Algebra: History and Structure, Parallel Matrix Multiplication. James Demmel www.cs.berkeley.edu/~demmel/cs267_Spr09. Outline. History and motivation Structure of the Dense Linear Algebra motif Parallel Matrix-matrix multiplication

tokala
Download Presentation

CS 267 Dense Linear Algebra: History and Structure, Parallel Matrix Multiplication

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 267Dense Linear Algebra:History and Structure,Parallel Matrix Multiplication James Demmel www.cs.berkeley.edu/~demmel/cs267_Spr09 CS267 Lecture 10

  2. Outline • History and motivation • Structure of the Dense Linear Algebra motif • Parallel Matrix-matrix multiplication • Parallel Gaussian Elimination (next time) CS267 Lecture 10

  3. Outline • History and motivation • Structure of the Dense Linear Algebra motif • Parallel Matrix-matrix multiplication • Parallel Gaussian Elimination (next time) CS267 Lecture 10

  4. Motifs The Motifs (formerly “Dwarfs”) from “The Berkeley View” (Asanovic et al.) Motifs form key computational patterns

  5. A brief history of (Dense) Linear Algebra software (1/6) • In the beginning was the do-loop… • Libraries like EISPACK (for eigenvalue problems) • Then the BLAS (1) were invented (1973-1977) • Standard library of 15 operations (mostly) on vectors • “AXPY” ( y = α·x + y ), dot product, scale (x = α·x ), etc • Up to 4 versions of each (S/D/C/Z), 46 routines, 3300 LOC • Goals • Common “pattern” to ease programming, readability, self-documentation • Robustness, via careful coding (avoiding over/underflow) • Portability + Efficiency via machine specific implementations • Why BLAS 1 ? They do O(n1) ops on O(n1) data • Used in libraries like LINPACK (for linear systems) • Source of the name “LINPACK Benchmark” (not the code!) CS267 Lecture 10

  6. Current Records for Solving Dense Systems (11/2008) www.netlib.org, click on Performance Database Server Gigaflops Machine n=100 n=1000 Any n Peak “Roadrunner” (LANL) 1.1M 1.5M (1.1 Petaflops, 130K cores, 2.5 MW, n=2.3M) NEC SX 8 (fastest in 2005) (8 proc, 2 GHz) 75.1 128 (1 proc, 2 GHz) 2.2 15.0 16 … Palm Pilot III .00000169 (1.69 Kiloflops) CS267 Lecture 10

  7. A brief history of (Dense) Linear Algebra software (2/6) • But the BLAS-1 weren’t enough • Consider AXPY ( y = α·x + y ): 2n flops on 3n read/writes • Computational intensity = (2n)/(3n) = 2/3 • Too low to run near peak speed (read/write dominates) • Hard to vectorize (“SIMD’ize”) on supercomputers of the day (1980s) • So the BLAS-2 were invented (1984-1986) • Standard library of 25 operations (mostly) on matrix/vector pairs • “GEMV”: y = α·A·x + β·x, “GER”: A = A + α·x·yT, x = T-1·x • Up to 4 versions of each (S/D/C/Z), 66 routines, 18K LOC • Why BLAS 2 ? They do O(n2) ops on O(n2) data • So computational intensity still just ~(2n2)/(n2) = 2 • OK for vector machines, but not for machine with caches CS267 Lecture 10

  8. A brief history of (Dense) Linear Algebra software (3/6) • The next step: BLAS-3 (1987-1988) • Standard library of 9 operations (mostly) on matrix/matrix pairs • “GEMM”: C = α·A·B + β·C, C = α·A·AT + β·C, B = T-1·B • Up to 4 versions of each (S/D/C/Z), 30 routines, 10K LOC • Why BLAS 3 ? They do O(n3) ops on O(n2) data • So computational intensity (2n3)/(4n2) = n/2 – big at last! • Good for machines with caches, other mem. hierarchy levels • How much BLAS1/2/3 code so far (all at www.netlib.org/blas) • Source: 142 routines, 31K LOC, Testing: 28K LOC • Reference (unoptimized) implementation only • Ex: 3 nested loops for GEMM • Lots more optimized code (eg Homework 1) • Motivates “automatic tuning” of the BLAS (details later) CS267 Lecture 10

  9. CS267 Lecture 8

  10. A brief history of (Dense) Linear Algebra software (4/6) • LAPACK – “Linear Algebra PACKage” - uses BLAS-3 (1989 – now) • Ex: Obvious way to express Gaussian Elimination (GE) is adding multiples of one row to other rows – BLAS-1 • How do we reorganize GE to use BLAS-3 ? (details later) • Contents of LAPACK (summary) • Algorithms we can turn into (nearly) 100% BLAS 3 • Linear Systems: solve Ax=b for x • Least Squares: choose x to minimize ||r||2S ri2where r=Ax-b • Algorithms we can only make up to 50% BLAS 3 (so far) • “Eigenproblems”: Findland x where Ax = l x • Singular Value Decomposition (SVD): ATAx=2x • Error bounds for everything • Lots of variants depending on A’s structure (banded, A=AT, etc) • How much code? (Release 3.2, Nov 2008) (www.netlib.org/lapack) • Source: 1582 routines, 490K LOC, Testing: 352K LOC • Ongoing development (at UCB and elsewhere) (class projects!) CS267 Lecture 10

  11. A brief history of (Dense) Linear Algebra software (5/6) • Is LAPACK parallel? • Only if the BLAS are parallel (possible in shared memory) • ScaLAPACK – “Scalable LAPACK” (1995 – now) • For distributed memory – uses MPI • More complex data structures, algorithms than LAPACK • Only (small) subset of LAPACK’s functionality available • Details later (class projects!) • All at www.netlib.org/scalapack CS267 Lecture 10

  12. Success Stories for Sca/LAPACK • Widely used • Adopted by Mathworks, Cray, Fujitsu, HP, IBM, IMSL, Intel, NAG, NEC, SGI, … • >96M web hits(in 2009, 56M in 2006) @ Netlib (incl. CLAPACK, LAPACK95) • New Science discovered through the solution of dense matrix systems • Nature article on the flat universe used ScaLAPACK • Other articles in Physics Review B that also use it • 1998 Gordon Bell Prize • www.nersc.gov/news/reports/newNERSCresults050703.pdf Cosmic Microwave Background Analysis, BOOMERanG collaboration, MADCAP code (Apr. 27, 2000). ScaLAPACK CS267 Lecture 10

  13. A brief history of (Dense) Linear Algebra software (6/6) • PLASMA and MAGMA (now) • Planned extensions to Multicore/GPU/Heterogeneous • Can one software infrastructure accommodate all algorithms and platforms of current (future) interest? • How much code generation and tuning can we automate? • Details later (Class projects!) • Other related projects • BLAST Forum (www.netlib.org/blas/blast-forum) • Attempt to extend BLAS to other languages, add some new functions, sparse matrices, extra-precision, interval arithmetic • Only partly successful (extra-precise BLAS used in latest LAPACK) • FLAME (www.cs.utexas.edu/users/flame/) • Formal Linear Algebra Method Environment • Attempt to automate code generation across multiple platforms CS267 Lecture 10

  14. Outline • History and motivation • Structure of the Dense Linear Algebra motif • Parallel Matrix-matrix multiplication • Parallel Gaussian Elimination (next time) CS267 Lecture 10

  15. What could go into the linear algebra motif(s)? For all linear algebra problems For all matrix/problem structures For all data types For all architectures and networks For all programming interfaces Produce best algorithm(s) w.r.t. performance and accuracy (including condition estimates, etc) Need to prioritize, automate!

  16. For all linear algebra problems:Ex: LAPACK Table of Contents • Linear Systems • Least Squares • Overdetermined, underdetermined • Unconstrained, constrained, weighted • Eigenvalues and vectors of Symmetric Matrices • Standard (Ax = λx), Generalized (Ax=λxB) • Eigenvalues and vectors of Unsymmetric matrices • Eigenvalues, Schur form, eigenvectors, invariant subspaces • Standard, Generalized • Singular Values and vectors (SVD) • Standard, Generalized • Level of detail • Simple Driver • Expert Drivers with error bounds, extra-precision, other options • Lower level routines (“apply certain kind of orthogonal transformation”)

  17. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general , pair • GT – tridiagonal • HB – Hermitian banded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  18. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general , pair • GT – tridiagonal • HB – Hermitian banded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  19. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general, pair • GT – tridiagonal • HB – Hermitianbanded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  20. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general, pair • GT – tridiagonal • HB – Hermitian banded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  21. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general, pair • GT – tridiagonal • HB – Hermitian banded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  22. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general , pair • GT – tridiagonal • HB – Hermitianbanded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  23. For all matrix/problem structures:Ex: LAPACK Table of Contents • BD – bidiagonal • GB – general banded • GE – general • GG – general, pair • GT – tridiagonal • HB – Hermitian banded • HE – Hermitian • HG – upper Hessenberg, pair • HP – Hermitian, packed • HS – upper Hessenberg • OR – (real) orthogonal • OP – (real) orthogonal, packed • PB – positive definite, banded • PO – positive definite • PP – positive definite, packed • PT – positive definite, tridiagonal • SB – symmetric, banded • SP – symmetric, packed • ST – symmetric, tridiagonal • SY – symmetric • TB – triangular, banded • TG – triangular, pair • TB – triangular, banded • TP – triangular, packed • TR – triangular • TZ – trapezoidal • UN – unitary • UP – unitary packed

  24. Organizing Linear Algebra – in books www.netlib.org/lapack www.netlib.org/scalapack gams.nist.gov www.netlib.org/templates www.cs.utk.edu/~dongarra/etemplates

  25. Outline • History and motivation • Structure of the Dense Linear Algebra motif • Parallel Matrix-matrix multiplication • Parallel Gaussian Elimination (next time) CS267 Lecture 10

  26. Different Parallel Data Layouts for Matrices (not all!) 1) 1D Column Blocked Layout 2) 1D Column Cyclic Layout 4) Row versions of the previous layouts b 3) 1D Column Block Cyclic Layout Generalizes others 6) 2D Row and Column Block Cyclic Layout 5) 2D Row and Column Blocked Layout CS267 Lecture 10

  27. Parallel Matrix-Vector Product • Compute y = y + A*x, where A is a dense matrix • Layout: • 1D row blocked • A(i) refers to the n by n/p block row that processor i owns, • x(i) and y(i) similarly refer to segments of x,y owned by i • Algorithm: • Foreach processor i • Broadcast x(i) • Compute y(i) = A(i)*x • Algorithm uses the formula y(i) = y(i) + A(i)*x = y(i) + Sj A(i,j)*x(j) P0 P1 P2 P3 x P0 P1 P2 P3 A(0) y A(1) A(2) A(3) CS267 Lecture 10

  28. Matrix-Vector Product y = y + A*x • A column layout of the matrix eliminates the broadcast of x • But adds a reduction to update the destination y • A 2D blocked layout uses a broadcast and reduction, both on a subset of processors • sqrt(p) for square processor grid P0 P1 P2 P3 P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 CS267 Lecture 10

  29. Parallel Matrix Multiply • Computing C=C+A*B • Using basic algorithm: 2*n3 Flops • Variables are: • Data layout • Topology of machine • Scheduling communication • Use of performance models for algorithm design • Message Time = “latency” + #words * time-per-word = a + n*b • Efficiency (in any model): • serial time / (p * parallel time) • perfect (linear) speedup efficiency = 1 CS267 Lecture 10

  30. p0 p1 p2 p3 p4 p5 p6 p7 Matrix Multiply with 1D Column Layout • Assume matrices are n x n and n is divisible by p • A(i) refers to the n by n/p block column that processor i owns (similiarly for B(i) and C(i)) • B(i,j) is the n/p by n/p sublock of B(i) • in rows j*n/p through (j+1)*n/p • Algorithm uses the formula C(i) = C(i) + A*B(i) = C(i) + Sj A(j)*B(j,i) May be a reasonable assumption for analysis, not for code CS267 Lecture 10

  31. Matrix Multiply: 1D Layout on Bus or Ring • Algorithm uses the formula C(i) = C(i) + A*B(i) = C(i) + Sj A(j)*B(j,i) • First consider a bus-connected machine without broadcast: only one pair of processors can communicate at a time (ethernet) • Second consider a machine with processors on a ring: all processors may communicate with nearest neighbors simultaneously CS267 Lecture 10

  32. MatMul: 1D layout on Bus without Broadcast Naïve algorithm: C(myproc) = C(myproc) + A(myproc)*B(myproc,myproc) for i = 0 to p-1 for j = 0 to p-1 except i if (myproc == i) send A(i) to processor j if (myproc == j) receive A(i) from processor i C(myproc) = C(myproc) + A(i)*B(i,myproc) barrier Cost of inner loop: computation: 2*n*(n/p)2 = 2*n3/p2 communication: a + b*n2 /p CS267 Lecture 10

  33. Naïve MatMul (continued) Cost of inner loop: computation: 2*n*(n/p)2 = 2*n3/p2 communication: a + b*n2 /p … approximately Only 1 pair of processors (i and j) are active on any iteration, and of those, only i is doing computation => the algorithm is almost entirely serial Running time: = (p*(p-1) + 1)*computation + p*(p-1)*communication  2*n3 + p2*a + p*n2*b This is worse than the serial time and grows with p. When might you still want to do this? CS267 Lecture 10

  34. Matmul for 1D layout on a Processor Ring • Pairs of adjacent processors can communicate simultaneously Copy A(myproc) into Tmp C(myproc) = C(myproc) + Tmp*B(myproc , myproc) for j = 1 to p-1 Send Tmp to processor myproc+1 mod p Receive Tmp from processor myproc-1 mod p C(myproc) = C(myproc) + Tmp*B( myproc-j mod p , myproc) • Same idea as for gravity in simple sharks and fish algorithm • May want double buffering in practice for overlap • Ignoring deadlock details in code • Time of inner loop = 2*(a + b*n2/p) + 2*n*(n/p)2 CS267 Lecture 10

  35. Matmul for 1D layout on a Processor Ring • Time of inner loop = 2*(a + b*n2/p) + 2*n*(n/p)2 • Total Time = 2*n* (n/p)2 + (p-1) * Time of inner loop •  2*n3/p + 2*p*a + 2*b*n2 • (Nearly) Optimal for 1D layout on Ring or Bus, even with Broadcast: • Perfect speedup for arithmetic • A(myproc) must move to each other processor, costs at least (p-1)*cost of sending n*(n/p) words • Parallel Efficiency = 2*n3 / (p * Total Time) = 1/(1 + a * p2/(2*n3) + b * p/(2*n) ) = 1/ (1 + O(p/n)) • Grows to 1 as n/p increases (or a and b shrink) CS267 Lecture 10

  36. MatMul with 2D Layout • Consider processors in 2D grid (physical or logical) • Processors can communicate with 4 nearest neighbors • Broadcast along rows and columns • Assume p processors form square s x s grid, s = p1/2 p(0,0) p(0,1) p(0,2) p(0,0) p(0,1) p(0,2) p(0,0) p(0,1) p(0,2) = * p(1,0) p(1,1) p(1,2) p(1,0) p(1,1) p(1,2) p(1,0) p(1,1) p(1,2) p(2,0) p(2,1) p(2,2) p(2,0) p(2,1) p(2,2) p(2,0) p(2,1) p(2,2) CS267 Lecture 10

  37. Cannon’s Algorithm … C(i,j) = C(i,j) + S A(i,k)*B(k,j) … assume s = sqrt(p) is an integer forall i=0 to s-1 … “skew” A left-circular-shift row i of A by i … so that A(i,j) overwritten by A(i,(j+i)mod s) forall i=0 to s-1 … “skew” B up-circular-shift column i of B by i … so that B(i,j) overwritten by B((i+j)mod s), j) for k=0 to s-1 … sequential forall i=0 to s-1 and j=0 to s-1 … all processors in parallel C(i,j) = C(i,j) + A(i,j)*B(i,j) left-circular-shift each row of A by 1 up-circular-shift each column of B by 1 k CS267 Lecture 10

  38. Cannon’s Matrix Multiplication C(1,2) = A(1,0) * B(0,2) + A(1,1) * B(1,2) + A(1,2) * B(2,2) CS267 Lecture 10

  39. Initial Step to Skew Matrices in Cannon • Initial blocked input • After skewing before initial block multiplies A(0,0) A(0,1) A(0,2) B(0,0) B(0,1) B(0,2) A(1,0) A(1,1) A(1,2) B(1,0) B(1,1) B(1,2) A(2,0) A(2,1) A(2,2) B(2,0) B(2,1) B(2,2) A(0,0) A(0,1) A(0,2) B(0,0) B(1,1) B(2,2) A(1,1) A(1,2) A(1,0) B(1,0) B(2,1) B(0,2) A(2,2) A(2,0) A(2,1) B(2,0) B(0,1) B(1,2) CS267 Lecture 10

  40. A(0,0) A(0,1) A(0,2) B(0,0) B(1,1) B(2,2) A(1,1) A(1,2) A(1,0) B(1,0) B(2,1) B(0,2) A(2,2) A(2,0) A(2,1) B(2,0) B(0,1) B(1,2) Skewing Steps in CannonAll blocks of A must multiply all like-colored blocks of B • First step • Second • Third A(0,2) A(0,0) A(0,1) B(1,0) B(2,1) B(0,2) A(1,0) A(1,1) B(2,0) B(0,1) B(1,2) A(1,2) A(2,0) A(2,2) B(0,0) B(1,1) B(2,2) A(2,1) A(0,0) B(2,0) B(0,1) B(1,2) A(0,2) A(0,1) A(1,0) B(0,0) B(1,1) B(2,2) A(1,1) A(1,2) A(2,0) A(2,1) A(2,2) B(1,0) B(2,1) B(0,2) CS267 Lecture 10

  41. Cost of Cannon’s Algorithm forall i=0 to s-1 … recall s = sqrt(p) left-circular-shift row i of A by i … cost ≤ s*(a + b*n2/p) forall i=0 to s-1 up-circular-shift column i of B by i … cost ≤ s*(a + b*n2/p) for k=0 to s-1 forall i=0 to s-1 and j=0 to s-1 C(i,j) = C(i,j) + A(i,j)*B(i,j) … cost = 2*(n/s)3 = 2*n3/p3/2 left-circular-shift each row of A by 1 … cost = a + b*n2/p up-circular-shift each column of B by 1 … cost = a + b*n2/p • Total Time = 2*n3/p + 4*s* + 4**n2/s • Parallel Efficiency = 2*n3 / (p * Total Time) • = 1/( 1 + a * 2*(s/n)3 + b * 2*(s/n) ) • = 1/(1 + O(sqrt(p)/n)) • Grows to 1 as n/s = n/sqrt(p) = sqrt(data per processor) grows • Better than 1D layout, which had Efficiency = 1/(1 + O(p/n)) CS267 Lecture 10

  42. Cannon’s Algorithm is “optimal” • Optimal means • Considering only O(n3) matmul algs (not Strassen) • Considering only O(n2/p) storage per processor • Use Irony/Toledo/Tiskin result from Lecture 2 • A processor doing n3flops from matmul with a fast memory of size M must do  (n3 / M1/2 ) references to slow memory • Same proof: a processor doing f flops from matmul with a local memory of size M must do  (f / M1/2 ) references to remote memory • Local memory = O(n2/p) , f  n3/p for at least one proc. • So f / M1/2 =  ((n3/p)/ (n2/p)1/2 ) =  ( n2/p1/2 ) • #Messages =  ( #words sent / max message size) =  ( (n2/p1/2)/(n2/p)) =  (p1/2) CS267 Lecture 10

  43. Pros and Cons of Cannon • So what if it’s “optimal”, is it fast? • Local computation one call to (optimized) matrix-multiply • Hard to generalize for • p not a perfect square • A and B not square • Dimensions of A, B not perfectly divisible by s=sqrt(p) • A and B not “aligned” in the way they are stored on processors • block-cyclic layouts • Memory hog (extra copies of local matrices) CS267 Lecture 10

  44. SUMMA Algorithm • SUMMA = Scalable Universal Matrix Multiply • Slightly less efficient, but simpler and easier to generalize • Presentation from van de Geijn and Watts • www.netlib.org/lapack/lawns/lawn96.ps • Similar ideas appeared many times • Used in practice in PBLAS = Parallel BLAS • www.netlib.org/lapack/lawns/lawn100.ps CS267 Lecture 10

  45. SUMMA B(k,j) k j k * = C(i,j) i A(i,k) • i, j represent all rows, columns owned by a processor • k is a block of b  1 rows or columns • C(i,j) = C(i,j) + SkA(i,k)*B(k,j) • Assume a pr by pc processor grid (pr = pc = 4 above) • Need not be square CS267 Lecture 10

  46. SUMMA B(k,j) k j k * = C(i,j) i A(i,k) For k=0 to n-1 … or n/b-1 where b is the block size … = # cols in A(i,k) and # rows in B(k,j) for all i = 1 to pr… in parallel owner of A(i,k) broadcasts it to whole processor row for all j = 1 to pc… in parallel owner of B(k,j) broadcasts it to whole processor column Receive A(i,k) into Acol Receive B(k,j) into Brow C_myproc = C_myproc + Acol * Brow CS267 Lecture 10

  47. SUMMA performance • To simplify analysis only, assume s = sqrt(p) For k=0 to n/b-1 for all i = 1 to s … s = sqrt(p) owner of A(i,k) broadcasts it to whole processor row … time = log s *( a + b * b*n/s), using a tree for all j = 1 to s owner of B(k,j) broadcasts it to whole processor column … time = log s *( a + b * b*n/s), using a tree Receive A(i,k) into Acol Receive B(k,j) into Brow C_myproc = C_myproc + Acol * Brow … time = 2*(n/s)2*b • Total time = 2*n3/p + a * log p * n/b + b * log p * n2 /s CS267 Lecture 10

  48. SUMMA performance • Total time = 2*n3/p + a * log p * n/b + b * log p * n2 /s • Parallel Efficiency = • 1/(1 + a * log p * p / (2*b*n2) + b * log p * s/(2*n) ) • Same b term as Cannon, except for log p factor • log p grows slowly so this is ok • Latency (a) term can be larger, depending on b • When b=1, get a * log p * n • As b grows to n/s, term shrinks to • a * log p * s (log p times Cannon) • Temporary storage grows like 2*b*n/s • Can change b to tradeoff latency cost with memory CS267 Lecture 10

  49. PDGEMM = PBLAS routine for matrix multiply Observations: For fixed N, as P increases Mflops increases, but less than 100% efficiency For fixed P, as N increases, Mflops (efficiency) rises DGEMM = BLAS routine for matrix multiply Maximum speed for PDGEMM = # Procs * speed of DGEMM Observations (same as above): Efficiency always at least 48% For fixed N, as P increases, efficiency drops For fixed P, as N increases, efficiency increases CS267 Lecture 8

  50. Summary of Parallel Matrix Multiplication • 1D Layout • Bus without broadcast - slower than serial • Nearest neighbor communication on a ring (or bus with broadcast): Efficiency = 1/(1 + O(p/n)) • 2D Layout • Cannon • Efficiency = 1/(1+O(a * ( sqrt(p) /n)3+b* sqrt(p) /n)) – optimal! • Hard to generalize for general p, n, block cyclic, alignment • SUMMA • Efficiency = 1/(1 + O(a * log p * p / (b*n2) + b*log p * sqrt(p) /n)) • Very General • b small => less memory, lower efficiency • b large => more memory, high efficiency • Used in practice (PBLAS) CS267 Lecture 10

More Related