Algorithms: Design and Analysis Summer School 2013 at VIASM: Random Structures and Algorithms

1. Algorithms: Design and AnalysisSummer School 2013 at VIASM: Random Structures and Algorithms Lecture 3: Greedy algorithms Phan Thị Hà Dương

2. Lecture 3: Greedy Algorithms 0. Introduction 1. Minimum spanning trees 2. Single-source shortest paths 3* Huffman codes

3. 0. Introduction Algorithms for optimization problems typically go through a sequence of steps, with a set of choices at each step. For many optimization problems, using dynamic programming to determine the best choices is overkill; simpler, more efficient algorithms will do. A greedy algorithm always makes the choice that looks best at the moment. That is, it makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution.

4. Strategy Design greedy algorithms according to the following sequence of steps: • Cast the optimization problem as one in which we make a choice and are left with one sub problem to solve. • Prove that there is always an optimal solution to the original problem that makes the greedy choice, so that the greedy choice is always safe. • Demonstrate that, having made the greedy choice, what remains is a sub problem with the property that if we combine an optimal solution to the sub problem with the greedy choice we have made, we arrive at an optimal solution to the original problem.

5. A very simple example • Problem: Given an array of positive integers. Find the greatest element of this array. 2. Solution: - At the beginning, the max = 0 • At each step i, we consider max and A(i), if A(i) > max then Max = A(i). • After n steps, Max is the greatest element of the array. 3. Greedy idea: At each step, we take the best solution for Max, After step I, Max is the greatest element of I first elements. The local solution implies a global solution for the problem.

6. 1. Minimum spanning trees Overview: • Problem: In the design of electronic circuitry, we wish to make the pins of several components electrically equivalent by wiring them together. Of all such arrangements, the one that uses the least amount of wire is usually the most desirable. • Model: undirected graph G = (V, E), V: the set of pins, E: the set of interconnections between pairs of pins, weight w(u, v): the cost to connect u and v. We wish to find an acyclic subset T of E that connects all of the vertices and whose total weight is minimized.

7. minimum-spanning-tree problem • T is acyclic and connects all of the vertices, it must form a tree, which we call a spanning tree since it "spans" the graph G. We call the problem of determining the tree T the minimum-spanning-tree problem. • Problem: given an weighted graph. Find a spanning tree with smallest weight.

8. 1. 1. Greedy generic algorithm GENERIC-MST(G, w) 1 A←Ø 2 while A does not form a spanning tree 3 do find an edge (u, v) that is safe for A 4 A ← A U {(u, v)} 5 return A We use the loop invariant as follows: •Initialization: After line 1, the set A trivially satisfies the loop invariant. •Maintenance: The loop in lines 2-4 maintains the invariant by adding only safe edges. •Termination: All edges added to A are in a minimum spanning tree, and so the set A is returned in line 5 must be a minimum spanning tree.

9. 1. 2. Kruskal’s algorithm • Each time, we have a forest. • At the beginning, it is a forest of n trees, each tree is a vertex. • At each step, we chose a smallest edge (u,v) that connect two different trees. • At the end, every trees are connected, we obtain an unique (spanning) tree.

10. MST-KRUSKAL MST-KRUSKAL(G, w) 1 A←Ø 2 for each vertex v in V[G] 3 do MAKE-SET(v) 4 sort the edges of E into nondecreasing order by weight w 5 for each edge (u, v) in E, taken in nondecreasing order by weight 6 do if FIND-SET(u) <> FIND-SET(v) 7 then A ← A U {(u, v)} 8 UNION(u, v) 9 return A

11. Analyzing of Kruskal’s algorithm • Line 2-3: O(n) • Line 4: O(E lg E) • FIND-SET = O(lg V) • Line 5-8: O(E lg V) • O(lg V) = O(lg E) • Total: O(E lg V)

12. 1. 3. Prim's algorithm We construct the tree step by step by adding one safe vertex and edge. • At the beginning, B contains one vertex and A has no edge. • At each step, A is the MST of B. Chose a smallest edge (u,v) such that u in V-B and v in B. Add u in B and Add (u,v) in A. • At the end, B = V, and A is MST of V.

13. Exercises • Write the Prim’s algorithm. • Analyze the Prim’s complexity. • What is the result of Kruskal’s algorithm if the graph is not connect ? • What is the result of Prim’s algorithm if the graph is not connect ?

14. 2. Single-Source Shortest Paths Problem: Given a Graph where each edge having a length. Given a source vertex s. Find shortest path from s to each vertex of G.

15. Variants • Single-destination shortest-paths problem: Find a shortest path to a given destination vertex t from each vertex v. By reversing the direction of each edge in the graph, we can reduce this problem to a single-source problem. • Single-pair shortest-path problem: Find a shortest path from u to v for given vertices u and v. If we solve the single-source problem with source vertex u, we solve this problem also. Moreover, no algorithms for this problem are known that run asymptotically faster than the best single-source algorithms in the worst case. • All-pairs shortest-paths problem: Find a shortest path from u to v for every pair of vertices u and v. Although this problem can be solved by running a single-source algorithm once from each vertex, it can usually be solved faster.

16. Idea of Dijkstra’s algorithm • Set S contains vertices determining shortest path from s. Q = V-S. • At the beginning: S=empty • At each step, chose v in Q with minimum distance to s. Add v to S (delete v from Q). Update the distance to s for all neighbors of v. • At the end, S=V

17. Lemma: a sub path of a shortest path is a shortest path. • Corollary: There exists an spanning tree of V rooted at s such that for all vertex v, the unique path from s to v in the tree is a shortest path.

18. Parameters of the algorithm • Array d where d(v) is the temporal distance from s to v. • Array Adj: Adj(v) is the list of neighbor of v. • Array w: w(u,v) the length of edge (u,v). • Array p: p(v) is the temporal parent of v. • Result: a tree rooted as s, and p(v) is the parent of v in the tree

19. Dijkstra’s algorithm Dijkstra (G,w,s) 1 for each v in V do d(v) = ∞; p(u) = Nil; d(s) = 0 2-3 S= Ø; Q = V 4 while Q <> Ø 5 do chose u in Q such that d(u) min 6 S = S U {u}; Q = Q –{u} 7 for each v in Adj(u) 8 do if d(v) > d(u) + w(u,v) then d(v) = d(u)+w(u,v); p(v) = u

20. Proof of Dijkstra’s algorithm We use the following loop invariant: At the start of each iteration of the while loop of lines 4-8, d[v] is the distance from s to v for each vertex v in S. Initialization: S = Ø, the invariant is trivially true. Maintenance: We wish to show that in each iteration, d[u] = distance(s,u) for the vertex added to set S. Termination: At termination, Q = Ø which, along with our earlier invariant that Q = V - S, implies that S = V. Thus, d[u] = distance(s,u) for all vertices u in V.

21. Maintenance We prove by recurrence that: • If u in S: d(u) = distance (s,u) • If u in Q: d(u) is the length of shortest path from s to u by using only vertices of S. • At the end, S= V, then d(u) is the real distance from s to u

22. Analyzing the Dijkstra’s algorithm • In general O(n^2) • If the graph is very sparse, we can realize q as a heap, each time we take u of Q such that d(u) min, the time is O(lg V). The number of operation is O(E) The total time is O(E lg V)

23. Traversal Salesman Problem • Problem: Given an weighted graph. Find a simple circuit passing each vertex exactly once (a Hamilton circuit) with minimum weight. • Greedy algorithms ? • This is a classic example such that the greedy algorithm does not work.

24. Counter example j = 1 2 3 4 5 i= 1 3 10 11 7 25 2 6 12 8 26 3 9 4 20 4 5 15 5 18

25. Chosen edges: 12, 35, 45, 23, 46, 16 Circuit: (1,2,3,5,4,6,1), length = 58 Minimum length = 56

26. Conclusion • Greedy algorithms do not always yield optimal solutions, but for many problems they do. • In many case, the idea of a greedy algorithm is very natural and easy to realize. • The greedy method is quite powerful and works well for a wide range of problems.