Separatosr in the Real World Practice

1. Separatosr in the Real World Practice Slides combined from www.cs.berkeley.edu/~demmel/cs267_Spr07 www.cs.cmu.edu/~guyb/realworld/slidesF03/src/separators1.ppt 15-853

2. 2 0 1 3 6 4 5 7 8 Edge Separators • An edge separator : a set of edges E’ ½ E which partitions V into V1 and V2 • Criteria: |V1|, |V2| balanced |E’| is small V1 V2 E’ 15-853

3. Vertex Separators • An vertex separator : a set of vertices V’ ½ V which partitions V into V1 and V2 • Criteria: |V1|, |V2| balanced |V’| is small 2 0 V2 1 V1 3 6 4 5 7 8 15-853

4. Other names • Sometimes referred to as • graph partitioning (probably more common than “graph separators”) • graph bisectors • graph bifurcators • balanced graph cuts 15-853

5. Application • Telephone network design • Original application, algorithm due to Kernighan • Load Balancing while Minimizing Communication • Sparse Matrix times Vector Multiplication • Solving PDEs • N = {1,…,n}, (j,k) in E if A(j,k) nonzero, • WN(j) = #nonzeros in row j, WE(j,k) = 1 • VLSI Layout • N = {units on chip}, E = {wires}, WE(j,k) = wire length • Sparse Gaussian Elimination • Used to reorder rows and columns to increase parallelism, and to decrease “fill-in” • Data mining and clustering • Physical Mapping of DNA 15-853

6. 2 0 1 3 2 6 0 4 1 5 7 3 8 6 1 4 5 7 8 2 0 4 5 5 7 3 8 8 6 3 0 7 4 1 8 5 2 6 Recursive Separation 15-853

7. What graphs have small separators • Planar graphs: O(n1/2) vertex separators • 2d meshes, constant genus, excluded minors • Almost planar graphs: • the internet, power networks, road networks • Circuits • need to be laid out without too many crossings • Social network graphs: • phone-call graphs, link structure of the web, citation graphs, “friends graphs” • 3d-grids and meshes: O(n1/2) 15-853

8. What graphs don’t have small separatos • Expanders: • Can someone give a quick proof that they don’t have small separators • Hypercubes: • O(n) edge separators O(n/(log n)1/2) vertex separators • Butterfly networks: • O(n/log n) separators ? • It is exactly the fact that they don’t have small separators that make them useful. 15-853

9. Applications of Separators • Circuit Layout (dates back to the 60s) • VLSI layout • Solving linear systems (nested dissection) • n3/2 time for planar graphs • Partitioning for parallel algorithms • Approximations to certain NP hard problems • TSP, maximum-independent-set • Clustering and machine learning • Machine vision 15-853

10. More Applications of Separators • Out of core algorithms • Register allocation • Shortest Paths • Graph compression • Graph embeddings 15-853

11. Available Software • METIS: U. Minnessota • PARTY: University of Paderborn • CHACO: Sandia national labs • JOSTLE: U. Geenwich • SCOTCH: U. Bordeaux • GNU: Popinet • Benchmarks: • Graph Partitioning Archive 15-853

12. Different Balance Criteria • Bisectors: 50/50 • Constant fraction cuts: e.g. 1/3, 2/3 • Trading off cut size for balance: • min cut criteria: • min quotient separator: • All versions are NP-hard 15-853

13. Other Variants of Separators • k-Partitioning: • Might be done with recursive partitioning, but direct solution can give better answers. • Weighted: • Weights on edges (cut size), vertices (balance) • Hypergraphs: • Each edge can have more than 2 end points • common in VLSI circuits • Multiconstraint: • Trying to balance different values at the same time. 15-853

14. |C| = n/2 Edge vs. Vertex separators • If a class of graphs satisfies an f(n) edge-separator theorem then it satisfies an f(n) vertex-separator. • The other way is not true (unless degree is bounded) 15-853

15. 2 0 1 3 2 6 0 4 1 5 7 3 8 6 1 4 5 7 8 2 0 4 5 5 7 3 8 8 6 3 0 7 4 1 8 5 2 6 Separator Trees 15-853

16. Asymptotics • If S is a class of graphs closed under the subgraph relation, then • Definition: S satisfies a f(n) vertex-separator theorem if there are constants a < 1 and b > 0 so that for every G  S there exists a cut set C  V, with • |C| <b f(|G|) cut size • |A| < a |G|, |B| < a |G| balance • Similar definition for edge separators. 15-853

17. Separator Trees • Theorem: For S satisfying an (a,b) n1-e edge separator theorem, we can generate a perfectly balanced separator tree with separator size |C| < k b f(|G|). • Proof: by picture |C| = b n1-e(1 + a + a2 + …) = b n1-e(1/1-a) 15-853

18. Algorithms • All are either heuristics or approximations • Kernighan-Lin (heuristic) • Planar graph separators (finds O(n1/2) separators) • Geometric separators (finds O(n(d-1)/d) separators) • Spectral (finds O(n(d-1)/d) separators) • Flow techniques (give log(n) approximations) • Multilevel recursive bisection (heuristic, currently most practical) 15-853

19. root Partitioning via Breadth First Search • BFS identifies 3 kinds of edges • Tree Edges - part of T • Horizontal Edges - connect nodes at same level • Interlevel Edges - connect nodes at adjacent levels • No edges connect nodes in levels • differing by more than 1 (why?) • BFS partioning heuristic • N = N1 U N2, where • N1 = {nodes at level <= L}, • N2 = {nodes at level > L} • Choose L so |N1| close to |N2| BFS partition of a 2D Mesh using center as root: N1 = levels 0, 1, 2, 3 N2 = levels 4, 5, 6 CS267 Guest Lecture

20. Kernighan-Lin Heuristic • Local heuristic for edge-separators based on “hill climbing”. Will most likely end in a local-minima. • Two versions: • Original: takes n2 times per step • Fiduccia-Mattheyses: takes n times per step 15-853

21. High-level description for both • Start with an initial cut that partitions the vertices into two equal size sets V1 and V2 • Want to swap two equal sized sets X A and Y  B to reduce the cut size. C A B X Y Note that finding the optimal subsets X and Y solves the optimal separator problem, so it is NP hard. We want some heuristic that might help. 15-853

22. Some Terminology • C(A,B) : the weighted cut between A and B • I(v) : the number of edges incident on v that stay within the partition • E(v) : the number of edges incident on v that go to the other partition • D(v) : E(v) - I(v) • D(u,v) : D(u) + D(v) - 2 w(u,v) • the gain for swapping u and v A v C B u 15-853

23. Kernighan-Lin improvement step • KL(G,A0,B0) •  (u  A0, v  B0 ) • put (u,v) in a PQ based on D(u,v) • for k = 1 to |V|/2 (u,v) = max(PQ) (Ak,Bk) = (Ak-1,Bk-1) swap (u,v) delete u and v entries from PQ update D on neighbors (and PQ) • select Ak,Bk with best Ck • Note that can take backward steps (D(u,v) can be negative). A v C B u 15-853

24. Fiduccia-Mattheyses improvement step • FM(G,A0,B0) • u  A0 put u in PQA based on D(u) • v  B0 put v in PQB based on D(v) • for k = 1 to |V|/2 • u = max(PQA) • put u on B side and update D • v = max(PQb) • put v on A side and update D • select Ak,Bk with best Ck A v C B u 15-853

25. A Bad Example for KL or FM • Consider following graph with initial cut given in red. 2 2 2 1 1 1 KL (or FM) will start on one side of grid and flip pairs over moving across the grid until the whole thing is flipped. The cut will never be better than the original bad cut. 15-853

26. Performance in Practice • In general the algorithms do very well at smoothing a cut that is approximately correct. • Used by most separator packages either • to smooth final results • to smooth partial results during the algorithm • K/L tested on very small graphs (|N|<=360) and got convergence after 2-4 sweeps • For random graphs (of theoretical interest) the probability of convergence in one step appears to drop like 2-|N|/30 15-853

27. Introduction to Multilevel Partitioning • If we want to partition G(N,E), but it is too big to do efficiently, what can we do? • 1) Replace G(N,E) by a coarse approximation Gc(Nc,Ec), and partition Gc instead • 2) Use partition of Gc to get a rough partitioning of G, and then iteratively improve it • What if Gc still too big? • Apply same idea recursively

28. Multilevel Partitioning - High Level Algorithm (N+,N- ) = Multilevel_Partition( N, E ) … recursive partitioning routine returns N+ and N- where N = N+ U N- if |N| is small (1) Partition G = (N,E) directly to get N = N+ U N- Return (N+, N- ) else (2)Coarsen G to get an approximation Gc = (Nc, Ec) (3) (Nc+ , Nc- ) = Multilevel_Partition( Nc, Ec ) (4)Expand (Nc+ , Nc- ) to a partition (N+ , N- ) of N (5)Improve the partition ( N+ , N- ) Return ( N+ , N- ) endif (5) “V - cycle:” (2,3) How do we Coarsen? Expand? Improve? (4) (5) (2,3) (4) (5) (2,3) (4) (1)

29. Multilevel Kernighan-Lin • Coarsen graph and expand partition using maximal matchings • Improve partition using Kernighan-Lin

30. Maximal Matching: Example

31. Example of Coarsening

32. Coarsening using a maximal matching 1) Construct a maximal matching Em of G(N,E) for all edges e=(j,k) in Em 2) collapse matched nodes into a single one Put node n(e) in Nc W(n(e)) = W(j) + W(k) … gray statements update node/edge weights for all nodes n in N not incident on an edge in Em 3) add unmatched nodes Put n in Nc … do not change W(n) … Now each node r in N is “inside” a unique node n(r) in Nc … 4) Connect two nodes in Nc if nodes inside them are connected in E for all edges e=(j,k) in Em for each other edge e’=(j,r) or (k,r) in E Put edge ee = (n(e),n(r)) in Ec W(ee) = W(e’) If there are multiple edges connecting two nodes in Nc, collapse them, adding edge weights

33. Expanding a partition of Gc to a partition of G

34. Multilevel Spectral Bisection • Coarsen graph and expand partition using maximal independent sets • Improve partition using Rayleigh Quotient Iteration

35. Example of Coarsening - encloses domain Dk = node of Nc

36. Coarsening using Maximal Independent Sets … Build “domains” D(k) around each node k in Ni to get nodes in Nc … Add an edge to Ec whenever it would connect two such domains Ec = empty set for all nodes k in Ni D(k) = ( {k}, empty set ) … first set contains nodes in D(k), second set contains edges in D(k) unmark all edges in E repeat choose an unmarked edge e = (k,j) from E if exactly one of k and j (say k) is in some D(m) mark e add j and e to D(m) else if k and j are in two different D(m)’s (say D(mi) and D(mj)) mark e add edge (mi, mj) to Ec else if both k and j are in the same D(m) mark e add e to D(m) else leave e unmarked endif until no unmarked edges

37. Expanding a partition of Gc to a partition of G • Need to convert an eigenvector vc of L(Gc) to an approximate eigenvector v of L(G) • Use interpolation: For each node j in N ifj is also a node in Nc, then v(j) = vc(j) … use same eigenvector component else v(j) = average of vc(k) for all neighbors k of j in Nc end if endif

38. Example: 1D mesh of 9 nodes

39. Improve eigenvector: Rayleigh Quotient Iteration j = 0 pick starting vector v(0) … from expanding vc repeat j=j+1 r(j) = vT(j-1) * L(G) * v(j-1) … r(j) = Rayleigh Quotient of v(j-1) … = good approximate eigenvalue v(j) = (L(G) - r(j)*I)-1 * v(j-1) … expensive to do exactly, so solve approximately … using an iteration called SYMMLQ, … which uses matrix-vector multiply (no surprise) v(j) = v(j) / || v(j) || … normalize v(j) until v(j) converges … Convergence is very fast: cubic

40. Example of convergence for 1D mesh

41. Available Implementations • Multilevel Kernighan/Lin • METIS (www.cs.umn.edu/~metis) • ParMETIS - parallel version • Multilevel Spectral Bisection • S. Barnard and H. Simon, “A fast multilevel implementation of recursive spectral bisection …”, Proc. 6th SIAM Conf. On Parallel Processing, 1993 • Chaco (www.cs.sandia.gov/CRF/papers_chaco.html) • Hybrids possible • Ex: Using Kernighan/Lin to improve a partition from spectral bisection

42. Comparison of methods • Compare only methods that use edges, not nodal coordinates • CS267 webpage and KK95a (see below) have other comparisons • Metrics • Speed of partitioning • Number of edge cuts • Other application dependent metrics • Summary • No one method best • Multi-level Kernighan/Lin fastest by far, comparable to Spectral in the number of edge cuts • Spectral give much better cuts for some applications • Ex: image segmentation • See “Normalized Cuts and Image Segmentation” by J. Malik, J. Shi

43. Number of edges cut for a 64-way partition For Multilevel Kernighan/Lin, as implemented in METIS (see KK95a) Expected # cuts 6427 2111 1190 11320 3326 4620 1746 8736 2252 4674 7579 3D mesh 31805 7208 3357 67647 13215 20481 5595 47887 7856 20796 39623 # of Nodes 144649 15606 4960 448695 38744 74752 10672 267241 17758 76480 201142 # of Edges 1074393 45878 9462 3314611 993481 261120 209093 334931 54196 152002 1479989 # Edges cut 88806 2965 675 194436 55753 11388 58784 1388 17894 4365 117997 Graph 144 4ELT ADD32 AUTO BBMAT FINAN512 LHR10 MAP1 MEMPLUS SHYY161 TORSO Description 3D FE Mesh 2D FE Mesh 32 bit adder 3D FE Mesh 2D Stiffness M. Lin. Prog. Chem. Eng. Highway Net. Memory circuit Navier-Stokes 3D FE Mesh Expected # cuts for 64-way partition of 2D mesh of n nodes n1/2 + 2*(n/2)1/2 + 4*(n/4)1/2 + … + 32*(n/32)1/2 ~ 17 * n1/2 Expected # cuts for 64-way partition of 3D mesh of n nodes = n2/3 + 2*(n/2)2/3 + 4*(n/4)2/3 + … + 32*(n/32)2/3 ~ 11.5 * n2/3

44. Thank you!