Recursive & Dynamic Programming

1. Recursive & Dynamic Programming Level II, Term ii CSE – 243 Md. Monjur-ul-hasan Lecturer Dept of cse, cuet Email: mail@monjur-ul-hasan.info

2. Introduction to Recursion • "Normally", we have methods that call other methods. • For example, the main() method calls the square() method. • Recursive Method: • A recursive method is a method that calls itself. main() square() compute() Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

3. Why Recursive • In computer science, some problems are more easily solved by using recursive methods. • In this course, will see many of examples of this. • For example: • Traversing through directories of a file system. • Traversing through a tree of search results. • Some sorting algorithms recursively sort data • For today, we will focus on the basic structure of using recursive methods. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

4. Recursive Situation LIST • Consider the following list of numbers: 24, 88, 40, 37 • Such a list can be defined as follows: A LIST is a: number or a: number comma LIST • That is, a LIST is defined to be a single number, or a number followed by a comma followed by a LIST • The concept of a LIST is used to define itself Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

5. Recursive Situation • The recursive part of the LIST definition is used several times, terminating with the non-recursive part: number comma LIST 24 , 88, 40, 37 number comma LIST 88 , 40, 37 number comma LIST 40 , 37 number 37 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

6. Recursive Situation GCD • gcd(a,b) = a ; when b = gcd(b,a%b) ; otherwise • gcd(4032, 1272) = gcd(1272, 216) = gcd(216, 192) = gcd(192, 24) = gcd(24, 0) = 24. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

7. Recursive Situation Finding Power The power function, p(x,n)=xn, can be defined recursively: • This leads to an power function that runs in O(n) time (for we make n recursive calls). • We can do better than this, however. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

8. Recursive Situation Recursive Squaring • We can derive a more efficient linearly recursive algorithm by using repeated squaring: • For example, 24= 2(4/2)2 = (24/2)2 = (22)2 = 42 = 16 25= 21+(4/2)2 = 2(24/2)2 = 2(22)2 = 2(42) = 32 26= 2(6/ 2)2 = (26/2)2 = (23)2 = 82 = 64 27= 21+(6/2)2 = 2(26/2)2 = 2(23)2 = 2(82) = 128. • Better then this solution is also available • For Practice Try to solve (bn%P) in this same way for large number of n Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

9. Recursive Situation Reversing an Array Algorithm ReverseArray(A, i, j): Input: An array A and nonnegative integer indices i and j Output: The reversal of the elements in A starting at index i and ending at j if i < j then Swap A[i] and A[ j] ReverseArray(A, i + 1, j - 1) return Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

10. Recursive Definition • Base case: You must always have some base case which can be solved without recursion • Making Progress: For cases that are to be solved recursively, the recursive call must always be a case that makes progress toward the base case. • Design Rule: Assume that the recursive calls work. • Compound Interest Rule: Never duplicate work by solving the same instance of a problem in separate recursive calls. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

11. Visualizing Recursion Example recursion trace: • Recursion trace • A box for each recursive call • An arrow from each caller to callee • An arrow from each callee to caller showing return value final answer return 3 * 2 = 6 call recursiveFactorial ( 3 ) 1 return 2 * = 2 call recursiveFactorial ( 2 ) return 1 * 1 = 1 call recursiveFactorial ( 1 ) Return 1 call recursiveFactorial ( 0 ) Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

12. Visualizing Recursion fact(1) 1 fact(2) fact(2) fact(2) 2 fact(3) fact(3) fact(3) fact(3) fact(3) 6 main() main() main() main() main() main() Time 7: Pop: fact(3) returns 6. Time 3: Push: fact(2) Time 4: Push: fact(1) Time 2: Push: fact(3) Time 6: Pop: fact(2) returns 2. Time 5: Pop: fact(1) returns 1. Inside findFactorial(1): if (number <= 1) return 1; else return (1 * factorial (0)); Inside findFactorial(3): if (number <= 1) return 1; else return (3 * factorial (2)); Inside findFactorial(2): if (number <= 1) return 1; else return (2 * factorial (1)); Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

13. Several Types of Recursion • Liner Recursion • May be more than one recursive method available but for each recursion it calls only one among them • Tail Recursion • The recursive call will be the last statement of the method • Chain Recursion • Function A will call Function B. function B will call Function C …… . … last function will call function A again. • Multiple Recursion • f(x) = a when x = 0 = b when x = 1 = a*f(x-1) + b*f(x-2) otherwise Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

14. Defining Arguments for Recursion • In creating recursive methods, it is important to define the methods in ways that facilitate recursion. • This sometimes requires we define additional paramaters that are passed to the method. • For example, we defined the array reversal method as ReverseArray(A, i, j), not ReverseArray(A). To verify that a recursive definition works: • convince yourself that the base case(s) are handled correctly • ASSUME RECURSIVE CALLS WORK ON SMALLER PROBLEMS, then convince yourself that the results from the recursive calls are combined to solve the whole Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

15. When recursion? • when it is the most natural way of thinking about & implementing a solution • can solve problem by breaking into smaller instances, solve, combine solutions • when it is roughly equivalent in efficiency to an iterative solution • OR • when the problems to be solved are so small that efficiency doesn't matter • think only one level deep • make sure the recursion handles the base case(s) correctly • assume recursive calls work correctly on smaller problems • make sure solutions to the recursive problems are combined correctly • avoid infinite recursion • make sure there is at least one base case & each recursive call gets closer Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

16. Alternative of Recursive • any recursive method/class can be rewritten iteratively (i.e., using a loop) • but sometimes, a recursive definition is MUCH clearer e.g., PermutationGenerator would be very difficult to conceptualize & implement without recursion • Use stack to model the recursive into iterative method. [ use the data structure course concept] • When overhead is low it is always better to use recursive but in other case it is better to use iterative solution. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

17. Recursion pros and cons • All recursive solutions can be implemented without recursion. • Recursion is "expensive". The expense of recursion lies in the fact that we have multiple activation frames and the fact that there is overhead involved with calling a method. • If both of the above statements are true, why would we ever use recursion? • In many cases, the extra "expense" of recursion is far outweighed by a simpler, clearer algorithm which leads to an implementation that is easier to code. • Ultimately, the recursion is eliminated when the compiler creates assembly language (it does this by implementing the stack). Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

18. Think Your Own Recursion • Try to solve the problem for base case • Think in the middle of the problem with following criteria: • Define that with same problem but with different domain • The change in domain should leads that to the base case. • You need not to think all the problem together. Just think one step from middle and the base case. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

19. Generating Permutation Systematically numerous applications require systematically generating permutations (orderings) • e.g., word games, debugging concurrent systems, tournament pairings, . . . • want to be able to take some sequence of items (say a String of characters) and generate every possible arrangement without duplicates • "123"  "123", "132", "213", "231", "312", "321" • "tape"  "tape", "taep", "tpae", "tpea", "teap", "tepa", • "atpe", "atep", "apte", "apet", "aetp", "aept", • "ptae", "ptea", "pate", "paet", "peta", "peat", • "etap", "etpa", "eatp", "eapt", "epta", "epat“ • how do we generate permutations systematically? • must maintain initial ordering • must get all permutations, with no duplicates Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

20. Generating Permutation Systematically • Generate all permutations that start with ‘t' , then 'a' then ‘p‘ then ‘e’ • To generate permutations starting with ‘t', we need to find all permutations of "ape" • This is the same problem with simpler inputs. • Use recursion To get your permutation go: http://home.att.net/~srschmitt/script_permutations.html Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

21. All Permutation Algorithm consider P is a global array contain the string exch (int i, int j) { int t = p[i]; p[i] = p[j]; p[j] = t; } generate(int N) { int c; if (N == 1) doit(); for (c = 1; c <= N; c++) { exch(c, N); generate(N-1); exch(c, N); } } • Invoke by calling • generate(N); Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

22. Challenge to Students (75 Marks) • Redesign the algorithm with no global variable • Redesign the algorithm that u develop in 1 to generate permutation which contain k element from the n length string • Redesign the algorithm of describe in 1 to generate all combination from a string by taking k element each time? • Design a correct algorithm to generate permutation when the string contain repetitive character in the given string • Design a correct Algorithm to generate Combination of k element among n element where the string contain repetitive character. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

23. Marks • Fail to submit: -15 from your Lab work • Duplicate submission : add absolute 0 with your lab work • Correct solution will add marks as follows • Q1: 5 • Q2 & Q2: 10+10 = 20 • Q4 20 • Q5 30 • Total 75 This mark will add to you lab result. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

24. Recursive to Iterative • What is the problems with this recursion? • Is there any direct solution available of this recursive function? • How can we test and find if any such available. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

25. Guess-and-Test Method • In the guess-and-test method, we guess a closed form solution and then try to prove it is true by induction: • Guess: T(n) < cn log n. • Wrong: we cannot make this last line be less than cnlogn Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

26. Guess-and-Test Method, Part 2 • Recall the recurrence equation: • Guess #2: T(n) < cn log2 n. • if c > b. • So, T(n) is O(n log2 n). • In general, to use this method, you need to have a good guess and you need to be good at induction proofs. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

27. Master Method • Many divide-and-conquer recurrence equations have the form: • The Master Theorem: Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

28. Master Method, Example 1 • The form: • The Master Theorem: • Example: • Solution: logba=2, so case 1 says T(n) is O(n2). Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

29. Master Method, Example 2 • The form: • The Master Theorem: • Example: • Solution: logba=1, so case 2 says T(n) is O(n log2 n). Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

30. Master Method, Example 3 • The form: • The Master Theorem: • Example: Solution: logba=0, so case 3 says T(n) is O(n logn). As f(n) = nlogn is asymptotically larger then nlogba =n Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

31. Divide & Conquer Algorithm • Divide problem into sub-problems • Conquer by solving sub-problems recursively. If the sub-problems are small enough, solve them in brute force fashion • Combine the solutions of sub-problems into a solution of the original problem (tricky part) Example sequence alignment Block Alignment Four-Russians speedup Constructing LCS in sub-quadratic time MergeSort Finding the middle point in the alignment matrix in linear space Linear space Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

32. Dynamic Programming fibonacci(5) fibonacci(4) + fibonacci(3) fibonacci(3) + fibonacci(2)fibonacci(2) + fibonacci(1) fibonacci(2) + fibonacci(1) • Dynamic Programming is an algorithm design technique for optimization problems: often minimizing or maximizing. • Like divide and conquer, DP solves problems by combining solutions to subproblems. • Unlike divide and conquer, subproblems are not independent. • Subproblems may share subsubproblems, • However, solution to one subproblem may not affect the solutions to other subproblems of the same problem. (More on this later.) Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

33. Dynamic Programming: How to think? • DP reduces computation by • Solving subproblems in a bottom-up fashion. • Storing solution to a subproblem the first time it is solved. • Looking up the solution when subproblem is encountered again. • Key: determine structure of optimal solutions Elements of Dynamic Programming • Optimal substructure • Overlapping subproblems • Memorization Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

34. Optimizing Substructure • Show that a solution to a problem consists of making a choice, which leaves one or more subproblems to solve. • Suppose that you are given this last choice that leads to an optimal solution. • Given this choice, determine which subproblems arise and how to characterize the resulting space of subproblems. • Show that the solutions to the subproblems used within the optimal solution must themselves be optimal. Usually use cut-and-paste. • Need to ensure that a wide enough range of choices and subproblems are considered. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

35. Optimal SubStructure • Optimal substructure varies across problem domains: • 1. How many subproblemsare used in an optimal solution. • 2. How many choices in determining which subproblem(s) to use. • Informally, running time depends on (# of subproblems overall)  (# of choices). • How many subproblems and choices do the examples considered contain? • Dynamic programming uses optimal substructure bottom up. • Firstfind optimal solutions to subproblems. • Thenchoose which to use in optimal solution to the problem. • Pieces of larger problem have a sequential dependency Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

36. Overlapping Subproblems • The space of subproblems must be “small”. • The total number of distinct subproblems is a polynomial in the input size. • A recursive algorithm is exponential because it solves the same problems repeatedly. • If divide-and-conquer is applicable, then each problem solved will be brand new. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

37. DP Approach • Forward Approach • Backward Approach • Note that if the recurrence relations are formulated using the forward approach then the relations are solved backwards . i.e., beginning with the last decision • On the other hand if the relations are formulated using the backward approach, they are solved forwards. You can also represent the problem by a multistage graph Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

38. A Simple DP Approach Fibonacci Numbers Int fib(int n) { static intknownFib[MAXFIB];int x; if(knownFib[n]==0) { if(n==1 || n==2) knownFib[n] = 1; else knownFib[n] = fib(n-1) + fib(n-2); } } Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

39. Matrix-chain multiplication • What will be the number of scalar multiplication in a matrix multiplication of a 3X5 and 5X 10 matrix? • What will be new matrix’s direction? • What is the number of scalar multiplication of the following matrices A1 X A2 X A3 X A4 Where, the damnations are 3x5, 5x4, 4x2 and 2x5 We have the following choice: ((A1 A2)  A3)  A4, # of scalar multiplications: 3 * 5 * 4 + 3 * 4 * 2 + 3 * 2 * 5 = 114 (A1 (A2 A3))  A4, # of scalar multiplications: 3 * 5 * 2 + 5 * 4 * 2 + 3 * 2 * 5 = 100 (A1 A2)  (A3 A4), # of scalar multiplications: 3 * 5 * 4 + 3 * 4 * 5 + 4 * 2 * 5 = 160 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

40. Recursive Function • Let m(i, j) denote the minimum cost for computing Ai  Ai+1  … Aj • Computation sequence : • Time complexity : O(n3) 8 -40 Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

41. Since subproblems overlap, we don’t use recursion. Instead, we construct optimal subproblems “bottom-up.” Ni,i’s are easy, so start with them. Then do length 2,3,… subproblems,and so on. Running time: O(n3) Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

42. Matrix Chan Multiplication • Algorithm matrixChain(S): • Input: sequence S of n matrices to be multiplied • Output: number of operations in an optimal paranethization of S • for i← 1 to n-1 do • Ni,i← 0 • for b ← 1 to n-1 do • for i← 0 to n-b-1 do • j ← i+b • Ni,j← +infinity • for k ← ito j-1 do • Ni,j← min{Ni,j, Ni,k+Nk+1,j +didk+1 dj+1} • Si,j← k //for which Ni,j Minimum Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

43. Matrix Chan Multiplication • Print the Optimal Structure OPTIMAL-MAT(s,i,j) {if(i==j) then print “A”I else print “(“ OPTIMAL-MAT(s,i,s[i,j) OPTIMAL-MAT(s,s[i,j]+1, j) print “)”; } Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

44. Longest Common Subsequence (LCS) • Problem:Given 2 sequences, X = x1,...,xm and Y = y1,...,yn, find a common subsequence whose length is maximum. springtime ncaa tournament basketball printing north carolina krzyzewski Subsequence need not be consecutive, but must be in order. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

45. LCS (Native Algorithm) • For every subsequence of X, check whether it’s a subsequence of Y . • Time:Θ(n2m). • 2msubsequences of X to check. • Each subsequence takes Θ(n)time to check: scan Y for first letter, for second, and so on. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

46. LCS: Optimal Substructure Theorem Let Z = z1, . . . , zk be any LCS of X and Y . 1. If xm= yn, then zk= xm= ynand Zk-1 is an LCS of Xm-1 and Yn-1. 2. If xmyn, then either zkxmand Z is an LCS of Xm-1 and Y . 3. or zkynand Z is an LCS of X and Yn-1. Notation: prefix Xi= x1,...,xiis the first iletters of X. This says what any longest common subsequence must look like; do you believe it? Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

47. Recursive Solution c[springtime, printing] c[springtim, printing] c[springtime, printin] [springti, printing] [springtim, printin] [springtim, printin] [springtime, printi] [springt, printing] [springti, printin] [springtim, printi] [springtime, print] Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

48. Memorization • Keep track of c[a,b] in a table of nm entries: • top/down • bottom/up Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

49. LCS: Length Finding Algorithm LCS-LENGTH (X, Y) • m← length[X] • n← length[Y] • for i← 1 to m • do c[i, 0] ← 0 • for j ← 0 to n • do c[0, j ] ← 0 • for i← 1 to m • do for j ← 1 to n • do if xi= yj • then c[i, j ] ← c[i1, j1] + 1 • b[i, j ] ← “ ” • else if c[i1, j ] ≥ c[i, j1] • then c[i, j ] ← c[i 1, j ] • b[i, j ] ← “↑” • else c[i, j ] ← c[i, j1] • b[i, j ] ← “←” • return c and b Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET

50. Md. Monjur-ul-hasan. Lecturer, Dept. of CSE, CUET