CS 3343: Analysis of Algorithms

1. CS 3343: Analysis of Algorithms Lecture 4: Sum of series, Analyzing recursive algorithms

2. Outline • Review of last lecture • Sum of series • Analyzing recursive algorithms

3. L’ Hopital’s rule Condition: If both lim f(n) and lim g(n) =  or 0 lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ n→∞

4. Stirling’s formula or (constant)

5. Properties of asymptotic notations • Textbook page 51 • Transitivity f(n) = (g(n)) and g(n) = (h(n)) => f(n) = (h(n)) (holds true for o, O, , and  as well). • Symmetry f(n) = (g(n)) if and only if g(n) = (f(n)) • Transpose symmetry f(n) = O(g(n)) if and only if g(n) = (f(n)) f(n) = o(g(n)) if and only if g(n) = (f(n))

6. logarithms • lg n = log2 n • ln n = loge n, e ≈ 2.718 • lgkn = (lg n)k • lg lg n = lg (lg n) = lg(2)n • lg(k) n = lg lg lg … lg n • lg24 = ? • lg(2)4 = ? • Compare lgkn vs lg(k)n?

7. Useful rules for logarithms • For all a > 0, b > 0, c > 0, the following rules hold • logba = logca / logcb = lg a / lg b • logban = n logba • blogba = a • log (ab) = log a + log b • lg (2n) = ? • log (a/b) = log (a) – log(b) • lg (n/2) = ? • lg (1/n) = ? • logba = 1 / logab

8. Useful rules for exponentials • For all a > 0, b > 0, c > 0, the following rules hold • a0 = 1 (00 = ?) • a1 = a • a-1 = 1/a • (am)n = amn • (am)n = (an)m • aman = am+n

9. More advanced dominance ranking

10. General plan for analyzing time efficiency of a non-recursive algorithm • Decide parameter (input size) • Identify most executed line (basic operation) • worst-case = average-case? • T(n) = i ti • T(n) = Θ (f(n))

11. Find the order of growth for sums • T(n) = i=1..n i = Θ (n2) • T(n) = i=1..n log (i) = ? • T(n) = i=1..n n / 2i = ? • T(n) = i=1..n 2i = ? • … • How to find out the actual order of growth? • Math… • Textbook Appendix A.1 (page 1058-60)

12. Arithmetic series • An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant. e.g.: 1, 2, 3, 4, 5 or 10, 12, 14, 16, 18, 20 • In general: Recursive definition Closed form, or explicit formula Or:

13. Sum of arithmetic series If a1, a2, …, an is an arithmetic series, then e.g. 1 + 3 + 5 + 7 + … + 99 = ? (series definition: ai = 2i-1) This is ∑i = 1 to 50 (ai)

14. Geometric series • A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant. e.g.: 1, 2, 4, 8, 16, 32 or 10, 20, 40, 80, 160 or 1, ½, ¼, 1/8, 1/16 • In general: Recursive definition Closed form, or explicit formula Or:

15. Sum of geometric series if r < 1 if r > 1 if r = 1

16. Sum of geometric series if r < 1 if r > 1 if r = 1

17. Important formulas

18. Sum manipulation rules Example:

19. Sum manipulation rules Example:

20. Examples • i=1..n n / 2i = n * i=1..n (½)i = ? • using the formula for geometric series: i=0..n (½)i = 1 + ½ + ¼ + … (½)n = 2 • Application: algorithm for allocating dynamic memories

21. Examples • i=1..n log (i) = log 1 + log 2 + … + log n = log 1 x 2 x 3 x … x n = log n! =  (n log n) • Application: algorithm for selection sort using priority queue

22. Problem of the day How do you find a coffee shop if you don’t know on which direction it might be?

23. Recursive definition of sum of series • T (n) = i=0..n i is equivalent to: T(n) = T(n-1) + n T(0) = 0 • T(n) = i=0..n ai is equivalent to: T(n) = T(n-1) + an T(0) = 1 Recurrence relation Boundary condition Recursive definition is often intuitive and easy to obtain. It is very useful in analyzing recursive algorithms, and some non-recursive algorithms too.

24. Analyzing recursive algorithms

25. Recursive algorithms • General idea: • Divide a large problem into smaller ones • By a constant ratio • By a constant or some variable • Solve each smaller onerecursively or explicitly • Combine the solutions of smaller ones to form a solution for the original problem Divide and Conquer

26. MERGE-SORTA[1 . . n] • If n = 1, done. • Recursively sort A[ 1 . . n/2 ] and A[ n/2+1 . . n ] . • “Merge” the 2 sorted lists. Merge sort Key subroutine:MERGE

27. Merging two sorted arrays Subarray 1 Subarray 2 20 13 7 2 12 11 9 1

28. Merging two sorted arrays Subarray 1 Subarray 2 20 13 7 2 12 11 9 1

29. Merging two sorted arrays 20 13 7 2 12 11 9 1

30. Merging two sorted arrays 20 13 7 2 12 11 9 1

31. Merging two sorted arrays 20 13 7 2 12 11 9 1 1

32. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 1

33. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 1 2

34. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 1 2

35. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 1 2 7

36. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 1 2 7

37. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 1 2 7 9

38. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 1 2 7 9

39. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 1 2 7 9 11

40. Merging two sorted arrays 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11

41. 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11 12 Merging two sorted arrays

42. How to show the correctness of a recursive algorithm? • By induction: • Base case: prove it works for small examples • Inductive hypothesis: assume the solution is correct for all sub-problems • Step: show that, if the inductive hypothesis is correct, then the algorithm is correct for the original problem.

43. MERGE-SORTA[1 . . n] • If n = 1, done. • Recursively sort A[ 1 . . n/2 ] and A[ n/2+1 . . n ] . • “Merge” the 2 sorted lists. Correctness of merge sort • Proof: • Base case: if n = 1, the algorithm will return the correct answer because A[1..1] is already sorted. • Inductive hypothesis: assume that the algorithm correctly sorts A[1.. n/2 ] and A[n/2+1..n]. • Step: if A[1.. n/2 ] and A[n/2+1..n] are both correctly sorted, the whole array A[1.. n/2 ] and A[n/2+1..n] is sorted after merging.

44. How to analyze the time-efficiency of a recursive algorithm? • Express the running time on input of size n as a function of the running time on smaller problems

45. Sloppiness:Should be T( n/2 ) + T( n/2) , but it turns out not to matter asymptotically. Analyzing merge sort T(n) Θ(1) 2T(n/2) f(n) MERGE-SORTA[1 . . n] • If n = 1, done. • Recursively sort A[ 1 . . n/2 ] and A[ n/2+1 . . n ] . • “Merge” the 2 sorted lists

46. Analyzing merge sort • Divide: Trivial. • Conquer: Recursively sort 2 subarrays. • Combine: Merge two sorted subarrays T(n) = 2T(n/2) + f(n) +Θ(1) # subproblems Dividing and Combining subproblem size • What is the time for the base case? • What is f(n)? • What is the growth order of T(n)? Constant

47. 20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11 12 Merging two sorted arrays Θ(n)time to merge a total of n elements (linear time).

48. T(n) = Θ(1)ifn = 1; 2T(n/2) + Θ(n)ifn > 1. Recurrence for merge sort • Later we shall often omit stating the base case when T(n) = Q(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence. • But what does T(n) solve to? I.e., is it O(n) or O(n2) or O(n3) or …?

49. Example: Find 9 3 5 7 8 9 12 15 Binary Search To find an element in a sorted array, we • Check the middle element • If ==, we’ve found it • else if less than wanted, search right half • else search left half

50. Example: Find 9 3 5 7 8 9 12 15 Binary Search To find an element in a sorted array, we • Check the middle element • If ==, we’ve found it • else if less than wanted, search right half • else search left half