3.3 Complexity of Algorithms

1. 3.3 Complexity of Algorithms • Linear Search Algorithm Determine if 3 is in the following lists A= { 1 4 8 -1 2 } B= { 3 4 8 6 5 } C= { 1 2 4 9 10 11 12 14 19 10 -11 0 } D= { 1 3 2 4 } For each list ( problem ), how much time will the algorithm take to finish the problem ?

2. 3.3 Complexity of Algorithms • Recall that the linear search algorithm go through the list from the beginning of the list until reaching the number ( 3 ) or the end of list. • Procedure linear search (x: integer a1, a2, . . . , an ) i=1; while ( i <= n and x =/= ai ) i := i+1 if i <= n, then location := i else location := 0

3. 3.3 Complexity of Algorithms A= { 1 4 8 -1 2 } ( exist at the end of the list) B= { 3 4 8 6 5 } (exist at the beginning of the list) C= { 1 2 4 9 10 11 12 14 19 10 -11 0 } D= { 1 3 2 4 } (exit after comparing two elements) List B will finish before lists A, B and C, even though list B is longer than D. List C will take the longest to finish.

4. 3.3 Complexity of Algorithms A= { 1 4 8 -1 2 } B= { 3 4 8 6 5 } C= { 1 2 4 9 10 11 12 14 19 10 -11 0 } D= { 1 3 2 4 } Clearly, the time the algorithm takes to finish • Depends on the size of the problem (list). • Even for problems of the same size, there are worst-case, best case, and average case.

5. 3.3 Complexity of Algorithms • Time Complexity: Analysis of the time required to solve a problem of a particular size. • Space Complexity: Analysis of the computer memory required to solve a problem of a particular size.

6. 3.3 Complexity of Algorithms • Describe the (worst-case) time complexity of the linear search algorithm. Procedure linear search (x: integer a1, a2, . . . , an ) i=1; while ( i <= n and x =/= ai ) i := i+1 if i <= n, then location := i else location := 0 The time complexity depends on the number of comparisons because they are the basic operations used. So how many comparisons are there if (in the worst-case) you have to go to the end of the list?

7. 3.3 Complexity of Algorithms • At each step of the loop, there are two comparisons. • Outside the loop, there is one more comparison. Therefore, if the list has n elements, we will have 2 n + 1 comparisons. In our machine, if each comparison takes k seconds (say, k = 0.00001), then for a list of size n, we’d expect that the algorithm to finish in k* (2 n + 1) seconds. This result is linear (degree one) in n. If list A is twice as long as list B, it will take twice as long to finish A than B. The complexity (in this case) is denote as O(n) (big-O of n, we will discuss this next week 3.2).

8. 3.3 Complexity of Algorithms Note that we will never know the exact time the algorithm will take to finish a given problem (that depends also on the machine you use). The main issue addressed by time complexity analysis is how does the algorithm’s performance scale with the size of the problem. The previous example O(n) shows a linear time complexity. There are other examples, such as O(n2), O( log n)….. If the time complexity is O(log n), you have to square n to double the running time (if you just double n, the difference in running time is almost negligible).

9. 3.3 Complexity of Algorithms • How about average complexity? Definition: the average number of operations used to solve the problem over all inputs of a given size. The time depends on where the element is located in the list. So given a list of size n. If x is in the first position in the list, it will take 3 comparisons. If it is in the second position, it will take 5 comparisons ….. So average is ( 3 + 5 + … + (2n+1) )/n = (2 ( 1+2+ … +n) +n)/n = n+2. Again, it depends linearly on n.

10. 3.3 Complexity of Algorithms • What is the worst-case complexity of bubble sort ?

11. 3.3 Complexity of Algorithms • Example • { 1 –11 50 6 8 –1} Use Bubble Sort (sort in increasing order} • After first pass • {-11 1 6 8 –1 50} (5 = 6 -1 comparisons) • After Second Pass • {-11 1 6 -1 8 50} (4 = 5 -1 comparisons) • After Third Pass • {-11 1 -1 6 8 50} (3 = 4 - 1 comparisons) • After Fourth Pass • {-11 -1 1 6 8 50} (2 = 3 – 1 comparisons) • After Fifth Pass • {-11 -11 6 8 50} (1 = 2 – 1 comparisons) • Done This list has 6 elements and there are 5 + 4 + 3 + 2 + 1 comparisons.

12. 3.3 Complexity of Algorithms • In general, if the list has n elements, we will have to do (n-1) + (n-2) …. + 2 +1 = (n-1) n / 2 comparisons. Notice that the complexity here depends quadratically (degree-2) in n. So if list A is twice as long as list B, it will take four times longer to process A than B.

13. 3.3 Complexity of Algorithms • How about insertion sort?

14. 3.3 Complexity of Algorithms • Example • { 1 –11 50 6 8 –1} Use Insertion Sort (sort in increasing order} • After first pass • {-11 1 50 6 8 -1} ( 1 comparison) • After Second Pass • {-11 1 50 6 8 -1} ( 2 comparisons) • After Third Pass • {--11 1 6 50 8 -1} ( 3 comparisons) • After Fourth Pass • {- 11 1 6 8 50 -1} ( 4 comparisons) • After Fifth Pass • {-11-11 6 8 50} ( 2 comparisons, worst-case 5 comparisons) • Done

15. 3.3 Complexity of Algorithms • In general, if the list has n elements, we will have to do 1+ 2+ … + (n-1) + (n-2) = (n-1) n / 2 comparisons (the result here is slightly different from the book). Notice that the complexity here depends quadratically (degree-2) in n. So if list A is twice as long as list B, it will take four times longer to process A than B.

16. Complexity Classes Linear search algorithm has linear complexity θ ( n ). Bubble sort has quadratic complexity θ ( n2 ). Binary search algorithm has logarithmic complexity θ ( log n ). The big-O and big- θ provide a language of describing the (time) complexity of algorithms.

17. Complexity Classes For problems with input of size n, the number of operations required by the algorithms. θ ( 1 ) Constant Complexity θ ( log n) Logarithmic Complexity θ ( n) Linear Complexity θ ( n log n) n log n Complexity θ ( nb ) Polynomial Complexity θ ( bn ) Exponential Complexity θ ( n! ) Factorial Complexity

18. Complexity Classes How do we known if a given type of problems is too difficult to solve with computers? Of course, some problems simply cannot be solved. A problem that is solvable using an algorithm with polynomial worst-case complexity is called tractable. algorithm will produce the solution to the problem for reasonably sized input in a relative short time. It is intractable if it cannot be solved using an algorithm with worst-case polynomial time complexity. Even for a small input, the algorithm will not produce the solution in a relative short time.