CSS342: Algorithm Analysis

CSS342: Algorithm Analysis Professor: Munehiro Fukuda CSS342: Algorithm Analysis

Today’s Topics • Mathematical analysis of algorithm complexity • Examples of algorithm complexity • Algorithm improvement from O(N3) to O(N) • Efficiency of search algorithms CSS342: Algorithm Analysis

Mathematical Analysis Measuring Algorithm Efficiency • Cost of a computer program • Human cost: Developing good problem-solving skills and programming style • Execution cost: Algorithm efficiency • Then, what in efficiency should we focus? Clever trick in coding? Execution time? • How should we compare programs? Based on a particular computer? What data should be given? CSS342: Algorithm Analysis

Mathematical Analysis Counting an Algorithm’s operations for ( i = 1; i <= n; i++ ) for ( j = 1; j <= i; j++ ) for ( k = 1; k <= 5; k++ ) task( ); Task: t operations ( or may take t seconds) Loop on k: 5 * t Loop on j: 5 * t (i = 1), 5 * t * 2 (i = 2), … 5 * t * n (i = n) n Loop on i: ∑(5 * t * i) = 5 * t * n * (n + 1) / 2 i=1 CSS342: Algorithm Analysis

Mathematical Analysis Algorithm Growth Rates • Nested loop i/j/k requires 5n(n+1)/2 time units to process a problem of size n. • Units may be seconds, minutes, and even dependent on what machines we use! • Nested loop i/j/k requires time proportional to 5n(n+1)/2 • Focus on how quickly the algorithm’s time requirement grows as a function of the problem size. Growth rate CSS342: Algorithm Analysis

Mathematical Analysis Order-of-Magnitude Analysis Let Algorithm A require time proportional to f(n). • A is of order at most g(n), O(g(n)) if f(n) <= k * g(n), for a positive constant k and n > n0 • A is of order at least g(n), Ω(g(n)) if f(n) >= k * g(n), for a positive constant k and n > n0 • A is of order g(n), Θ(g(n)) if f(n) = O(g(n)) and f(n) = Ω(g(n)) CSS342: Algorithm Analysis

Mathematical Analysis Example of O(g(n)), Ω(g(n)) ,andΘ(g(n)) Suppose Algorithm A requires time proportional to f(n) = n2 – 3n + 10 • f(n) < k* n2 , if k = 3 and n >= 2, and thus f(n) = O(n2 ) • f(n) > k * n2 , if k = 0.5 n>= 0, and thus f(n) = Ω(n2) • Therefore f(n) = Θ(n2) n 0.5n2 N2 -3N+10 3n2 0 0 10 0 1 0.5 8 3 2 2 8 12 3 4.5 10 27 CSS342: Algorithm Analysis

Mathematical Analysis Example of Program Analysis j = n while ( j >= 1 ) { for ( i = 1; i <= j; i++ ) x = x + 1; // How many times is x=x+1 executed? j = j / 2; } T(n): # x=x+1 executed In the 1st while-loop, j = n. for-loop executes x=x+1 n times. T(n)=Ω(n) In the 2nd while-loop, j <= n/2, n/2 times In the 3rd while-loop, j <= n/4, n/4 times In the kth while-loop, j <= n/2k-1, n/2k-1times CSS342: Algorithm Analysis

Mathematical Analysis Example of Program Analysis Cont’d Geometric Sum: a + ar1 + ar2 + …. + arn = a(rn+1 – 1)/(r – 1), where r != 1 (Proof is given in the week 4, “Induction”) Thus, T(n) <= n + n/2 + n/4 + … + n/2k-1 = n + n*(1/2) + n*(1/2)2 + … + n * (1/2)(k-1) = n((1/2)k-1)/(1/2 – 1) = n(1-1/2k)/(1-1/2) = 2n(1-1/2k) <= 2n T(n) = O(n) Therefore, T(n) = Θ(n) CSS342: Algorithm Analysis

Mathematical Analysis Focusing on Big O Notation • Upper-bound information is the main concern • In most cases, O(g(n))=Ω(g(n)) =Θ(g(n)) • O(1) < O(log2n) < O(n) < O(n*log2n) < O(n2) < O(n3) < O(2n) • Focus on the dominant factor: • Low-order terms can be ignored. O(n3+4n) = O(n3) • Multiplicative constant in the high-order term can be ignored. O(3n3+4) = O(n3) • O(f(n)) + O(g(n)) = O(f(n) + g(n)) • If f(n) = O(g(n)) and g(n) = O(h(n)) , then f(n)=O(h(n)) CSS342: Algorithm Analysis

Mathematical Analysis Intuitive interpretation of growth-rate functions O(1) Constant Independent of problem size O(log2n) Logarithmic Increase slowly as size increases. Ex. Binary search O(n) Linear Increase directly with size O(nlog2n) Increase more rapidly than a liner algorithm. Ex. Merge sort O(n2) Quadratic Ex. Two nested loops O(n3) Cubic Ex. Three nested loops O(2n) Exponential Increase too rapidly to be practical CSS342: Algorithm Analysis

Examples Examples of Algorithm ComplexityMinimum Element in an Array • Given an array of N items, find the smallest item. • What order (in big O) will this problem be bound to? 1 N 3 9 1 8 5 2 0 7 4 6 -1 10 -2 CSS342: Algorithm Analysis

Examples Examples of Algorithm ComplexityClosest Points in the Plane • Given N points in a plane (that is, an x-y coordinate system), find the pair of points that are closest together. • What order (in big O) will this problem be bound to? y 2,1 sqrt( (x2 – x1)2 + (y2 – y1)2 ) 1,4 1,6 4,5 4,7 5,4 5,6 6,3 6,7 7,1 x 8,4 CSS342: Algorithm Analysis

Examples Examples of Algorithm ComplexityColinear Points in the Plane • Given N points in a plane (that is, an x-y coordinate system), determine if any tree form a straight line. • What order (in big O) will this problem be bound to? y Find y = ax + b for two points. Check if the 3rd point satisfies this equation 2,1 1,4 1,6 4,5 4,7 5,4 5,6 6,3 6,7 7,1 x 8,4 CSS342: Algorithm Analysis

Algorithm Improvement The Maximum Contiguous Subsequence Sum Problem • Given a sequence of integers I1..IN, find and identify the sub sequence corresponding to the maximum value of ∑k=ijAk. N 1 -2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2 20 CSS342: Algorithm Analysis

We don’t need this loop! Simply increment j and add a new integer to the sum. Algorithm Improvement O(N3) Algorithm N 1 -2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2 i j j++ The most outer loop i++ The intermediate loop The most inner loop ∑k=ijAk int maxSum = 0; for ( int i = 0; i < n; i++ ) for ( int j = i; j < n; j++ ) { int thisSum = 0; for ( int k = i; k <= j; k++ ) thisSum += a[k]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } } CSS342: Algorithm Analysis

Algorithm Improvement O(N2) Algorithm N 1 -2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2 i j j++ The most outer loop i++ The intermediate loop ∑j=ijAj int maxSum = 0; for ( int i = 0; i < n; i++ ) { int thisSum = 0; for ( int j = i; j < n; j++ ) { thisSum += a[j]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } } } CSS342: Algorithm Analysis

Algorithm Improvement O(N) Algorithm Generally, the summation increases but if it gets below 0 ∑j=inAj 1 N ∑< 0 Ignore this part From here, restart the summation. int thisSum = 0; int maxSum = 0; for ( int i = 0, j = 0; j < n; j++ ) { thisSum += a[j]; if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } else if ( thisSun < 0 ) { i = j + 1; thisSum = 0; } } CSS342: Algorithm Analysis

Efficiency of Search Algorithms Worst, Best, and Average-case Analysis • Worst-case analysis: • Algorithm A requires no more than k * f(n) time units. • It might happen rarely, if at all, in practice. • Best-case analysis: • Like worst-case analysis, it might happen rarely. • Average-case analysis: • A requires no more than k * f(n) time units for all but a finite number of values of n • Difficulties: determining probability and distribution of size n and input data CSS342: Algorithm Analysis

Efficiency of Search Algorithms Efficiency of Sequential Search 29 10 14 37 13 52 46 75 5 43 21 69 88 1 Hit! Best case O(1) Hit! Worst case O(n) Hit! Average case O(n/2) = O(n) (Assume desired items are uniformly distributed.) CSS342: Algorithm Analysis

Efficiency of Search Algorithms Efficiency of Binary Search x high=15 low =0 mid = (15 + 0)/2 = 7 10 a 1 5 10 13 14 21 29 37 43 46 52 69 75 88 91 99 new high=7–1=6 template <class Comparable> int binarySearch( const vector<Comparable> &a, const Comparable &x ) { int low = 0; int high = a.size( ) – 1; int mid; while ( low <= high ) { mid = ( low + high ) / 2; if ( a[mid] < x ) low = mid + 1; else if ( a[mid] > x ) high = mid – 1; else return mid; } return NOT_FOUND; // NOT_FOUND = -1 } CSS342: Algorithm Analysis

Efficiency of Search Algorithms Efficiency of Binary Search 1 5 10 13 14 21 29 37 43 46 52 69 75 88 91 99 N = 2 K = 1 N = 4 K = 2 N = 8 K = 3 N = 16 K = 4 N = 2k K = K 2K-1 < N < 2K K – 1 < log2N < K K < 1 + log2N < K + 1 K = O(log2N) 1st step 2nd step 3rd step Hit! 4th step CSS342: Algorithm Analysis

Efficiency of Search Algorithms Efficiency of Interpolation Search • A way you start your search for “Hank Aaron” from the first 20 pages in a 500-page white page: next = low + ┌ (x – a[low]) / (a[high] – a[low]) * (high – low – 1)┐ Example: Assume low = 0, high = 999, a[low] = 1000, a[high]=1000000. If you search for the value 12000: next = 0 + ┌(12000 – 1000)/(1000000 – 1000) * (999 – 0 – 1) ┐ = 10 • Two assumptions: • Each access must be very expensive like a disk access. • Data must be uniformly distributed • Complexity • Average: O(log log N) • Worst: O(N) CSS342: Algorithm Analysis

Limitations Some Note for Algorithm Analysis • Checking an algorithm analysis • Empirically observe running time as increasing N. • Divide the time by O(f(N)). • The dividend should converge to a positive constant. • Limitations of Big-Oh Analysis • Does not work for a small N • Worst-case analysis may be uncommon • Average-case analysis is more difficult than worse case • Empirically observation depends on memory hierarchy and multiprogramming level, etc.. CSS342: Algorithm Analysis

Limitations Intractable, Unsolvable, and NP problems • Good algorithms: • Their worst-case time is proportional to a polynomial. • Intractable problems: • Their worst-case time cannot be bounded to a polynomial. • Unsolvable algorithms: • They have no algorithms at all. Ex. Halting problem • NP(Non-Polynomial) problems: • They are thought to be intractable, but not yet proved. • Ex1. Traveling salesperson problem • Ex2. Hamiltonian cycle CSS342: Algorithm Analysis

CSS342: Algorithm Analysis