UMass Lowell Computer Science 91.503 Analysis of Algorithms Prof. Giampiero Pecelli Fall, 2010

1. UMass Lowell Computer Science 91.503Analysis of AlgorithmsProf. Giampiero PecelliFall, 2010 Paradigms for Optimization Problems Dynamic Programming & Greedy Algorithms

2. Optimization This, generally, refers to classes of problems that possess multiple solutions at one level, and where we have a real-valued function defined on the solutions. Problem: find a solution that minimizes or maximizes the value of this function. Note: there is no guarantee that such a solution will be unique and, moreover, there is no guarantee that you will find it (local maxima, anyone?) unless the search is over a small enough search space or the function is restricted enough.

3. Optimization Question: are there classes of problems for which you can guarantee an optimizing solution can be found? Answer: yes. BUT you also need to find such a solution in a "reasonable" amount of time. We are going to look at two classes of problems, and the techniques that will succeed in constructing their solutions in a "reasonable" (i.e., low degree polynomial in the size of the initial data) amount of time.

4. Optimization We begin with a rough comparison that contrasts a method you are familiar with (divide and conquer) and the method (still unspecified) of Dynamic Programming (developed by Richard Bellman in the late 1940's and early 1950's). For some history and other ideas, see: http://en.wikipedia.org/wiki/Dynamic_programming

5. Two Algorithmic Models: Divide & Dynamic Conquer Programming View problem as collection of subproblems “Recursive” nature Independent subproblems Overlapping subproblems Number of subproblems depends on typically small partitioning factors Preprocessing Characteristic running time typically log depends on number function of n and difficulty of subproblems Primarily for optimization problems Optimal substructure: optimal solution to problem contains within it optimal solutions to subproblems

6. Dynamic Programming

7. Example: Rod Cutting (text) • You are given a rod of length n ≥ 0 (n in inches) • A rod of length i inches will be sold for pi dollars • Cutting is free (simplifying assumption) • Problem: given a table of prices pidetermine the maximum revenue rn obtainable by cutting up the rod and selling the pieces.

8. Example: Rod Cutting We can see immediately (from the values in the table) that n ≤ pn ≤ 3n. This is not very useful because: • The range of potential revenue is very large • Our finding quick upper and lower bounds depends on finding quickly the minimum and maximum pi/i ratios (one pass through the table), but then we are back to the point above….

9. Example: Rod Cutting Step 1: Characterizing an Optimal Solution • Question: in how many different ways can we cut a rod of length n? • For a rod of length 4: • 24 - 1 = 23 = 8 • For a rod of length n: 2n-1. Exponential: we cannot try all possibilities for n "large". The obvious exhaustive approach won't work.

10. Example: Rod Cutting Step 1: Characterizing an Optimal Solution • Question: in how many different ways can we cut a rod of length n? • Proof Details: a rod of length n can have exactly n-1 possible cut positions – choose 0 ≤ k ≤ n-1 actual cuts. We can choose the k cuts (without repetition) anywhere we want, so that for each such k the number of different choices is • When we sum up over all possibilities (k = 0 to k = n-1): • For a rod of length n: 2n-1.

11. Example: Rod Cutting Characterizing an Optimal Solution Let us find a way to solve the problem recursively (we might be able to modify the solution so that the maximum can be actually computed): assume we have cut a rod of length n into 0 ≤ k ≤ n pieces of lengthi1, …, ik, n = i1 +…+ ik, with revenue rn = pi1 + … + pik Assume further that this solution is optimal. How can we construct it? Advice: when you don’t know what to do next, start with a simple example and hope something will occur to you…

12. Example: Rod Cutting Characterizing an Optimal Solution We begin by constructing (by hand) the optimal solutions for i = 1, …, 10: r1 = 1 from sln. 1 = 1 (no cuts) r2 = 5 from sln. 2 = 2 (no cuts) r3 = 8 from sln. 3 = 3 (no cuts) r4 = 10 from sln. 4 = 2 + 2 r5 = 13 from sln. 5 = 2 + 3 r6 = 17 from sln. 6 = 6 (no cuts) r7 = 18 from sln. 7 = 1 + 6 or 7 = 2 + 2 + 3 r8 = 22 from sln. 8 = 2 + 6 r9 = 25 from sln. 9 = 3 + 6 r10 = 30 from sln. 10 = 10 (no cuts)

13. Example: Rod Cutting Characterizing an Optimal Solution Notice that in some cases rn = pn, while in other cases the optimal revenue rn is obtained by cutting the rod into smaller pieces. In ALL cases we have the recursion rn = max(pn, r1 + rn-1, r2 + rn-2, …, rn-1 + r1) exhibiting optimal substructure (meaning?) A slightly different way of stating the same recursion, which avoids repeating some computations, is rn = max1≤i≤n(pi + rn-i) And this latter relation can be implemented as a simple top-down recursive procedure:

14. Example: Rod Cutting Characterizing an Optimal Solution Time Out: How to justify the step from: rn = max(pn, r1 + rn-1, r2 + rn-2, …, rn-1 + r1) to rn = max1≤i≤n(pi + rn-i) Note: every optimal partitioning of a rod of length n has a first cut – a segment of, say, length i. The optimal revenue, rn, must satisfy rn = pi + rn-i, where rn-i is the optimal revenue for a rod of length n – i. If the latter were not the case, there would be a better partitioning for a rod of length n – i, giving a revenue r’n–i > rn-i and a total revenue r’n = pn + r’n-i > pi + rn-i = rn. Since we do not know which one of the leftmost cut positions provides the largest revenue, we just maximize over all the possible first cut positions.

15. Example: Rod Cutting Characterizing an Optimal Solution We can also notice that all the items we choose the maximum of are optimal in their own right: each substructure (max revenue for rods of lengths 1, …, n-1) is also optimal (again, optimal substructure property). Nevertheless, we are still in trouble: computing the recursion leads to recomputing a number (= overlapping subproblems) of values – how many?

16. Example: Rod Cutting Characterizing an Optimal Solution Let’s call Cut-Rod(p, 4), to see the effects on a simple case: The number of nodes for a tree corresponding to a rod of size n is:

17. Example: Rod Cutting Beyond Naïve Time Complexity • We have a problem: “reasonable size” problems are not solvable in “reasonable time” (but, in this case, they are solvable in “reasonable space”). • Specifically: • Note that navigating the whole tree requires 2n stack-frame activations. • Note also that no more than n + 1 stack-frames are active at any one time and that no more than n + 1 different values need to be computed or used. • Can we exploit these observations? • A standard solution method involves saving the values associated with each T(j), so that we compute each value only once (called “memoizing” = writing yourself a memo).

18. Example: Rod Cutting Naïve Caching We introduce two procedures:

19. Example: Rod Cutting More Sophisticated Caching We now remove some unnecessary complications:

20. Example: Rod Cutting Time Complexity Whether we solve the problem in a top-down or bottom-up manner the asymptotic time is Θ(n2), the major difference being recursive calls as compared to loop iterations. Why??

21. Example: Longest Common Subsequence (LCS): Motivation • Strand of DNA: string over finite set {A,C,G,T} • each element of set is a base: adenine, guanine, cytosine or thymine • Compare DNA similarities • S1 = ACCGGTCGAGTGCGCGGAAGCCGGCCGAA • S2 = GTCGTTCGGAATGCCGTTGCTCTGTAAA • One measure of similarity: • find the longest string S3 containing bases that also appear (not necessarily consecutively) in S1 and S2 • S3 = GTCGTCGGAAGCCGGCCGAA source: 91.503 textbook Cormen, et al.

22. Example: LCS Definitions source: 91.503 textbook Cormen, et al. • The sequence is a subsequence of if (strictly increasing indices of X) such that • example: is a subsequence of with index sequence • Z is common subsequence of X and Y if Z is subsequence of both X and Y • example: • common subsequence but not longest • common subsequence. Longest? Longest Common Subsequence Problem: Given 2 sequences X,Y, find maximum-length common subsequence Z.

23. Example: LCS Step 1: Characterize an LCS THM 15.1: Optimal LCS Substructure Given sequences: For any LCS of X and Y: 1 if xm = yn then zk = xm = ynand Zk-1 is an LCS of Xm-1 and Yn-1 2 if xm ≠ yn then zk ≠ xm Z is an LCS of Xm-1 and Y 3 if xm ≠ yn then zk ≠ yn Z is an LCS of X and Yn-1 PROOF: based on producing contradictions 1 a) Suppose zk ≠ xm. Appending xm = yn to Z contradicts longest nature of Z. b) To establish longest nature of Zk-1, suppose common subsequence W of Xm-1and Yn-1has length > k-1. Appending xm to W yields common subsequence of length > k= contradiction. 2 Common subsequence Wof Xm-1and Yof length > k would also be common subsequence of Xm, Y, contradicting longest nature of Z. 3 Similar to proof of (2) source: 91.503 textbook Cormen, et al.

24. ? Example: LCS Step 2: A Recursive Solution • Implications of Thm 15.1: no yes Find LCS(Xm-1, Yn-1) Find LCS(X, Yn-1) Find LCS(Xm-1, Y) LCS(X, Y) = LCS(Xm-1, Yn-1) + xm LCS(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1)) LCS(X, Y)

25. 0 if i=0 or j=0 c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj Example: LCS Step 2: A Recursive Solution (continued) source: 91.503 textbook Cormen, et al. • Overlapping subproblem structure: • Recurrence for length of optimal solution: Q(mn) distinct subproblems Conditions of problem can exclude some subproblems!

26. Example: LCS Step 3: Compute Length of an LCS c table (represent b table) source: 91.503 textbook Cormen, et al.

27. Example: LCS Step 4: Construct an LCS source: 91.503 textbook Cormen, et al.

28. Example: LCS Improve the Code source: 91.503 textbook Cormen, et al. • Can eliminate b table • c[i,j] depends only on 3 other c table entries: • c[i-1,j-1], c[i-1,j], c[i,j-1] • given value of c[i,j], can pick the one in O(1) time • reconstruct LCS in O(m+n) time similar to PRINT-LCS • same Q(mn) space, but Q(mn) was needed anyway... • Asymptotic space reduction • leverage: need only 2 rows of c table at a time • row being computed • previous row • can also do it with ~ space for 1 row of c table • but does not preserve LCS reconstruction data

29. Algorithmic Paradigm Context