Design and Analysis of Algorithms

1. Design and Analysis of Algorithms Dr. Aarij Mahmood Hussaan

2. Course Objective The last ½ century has witnessed the development of a beautiful and elegant new scientific field: the design and analysis of algorithms. This course will teach you how to “design” and “analyze” algorithms.

3. History • The name of “algorithm” comes from al-Khowarizmi, 9th century Persian mathematician and astronomer who wrote a book, eventually translated into Latin in the 12th century as Algoritmi de numeroIndorum ("al-Khowarizmi's book on Indian numbers"), that gave our modern pencil-and-paper algorithms for addition, subtraction, multiplication, and division. • Every algorithm should have: • Input, • Output, • Deterministic • Effective, • Finite.

4. Why study algorithms • Many aspects of “programming” are more important than algorithms: • user-friendliness, correctness, extensibility, maintainability, simplicity, scalability, cost, … • But good algorithm is fundamental to all of above.

5. Algorithm • A procedure for solving a (mathematical) problem in a finite number of steps that frequently involves repetition of an operation; broadly: a step-by-step method for accomplishing some task.

6. Algorithm: formal definition──Intro CS • A well-ordered collection of unambiguous and effectively computable operations that, when, executed, produces a result and halts in a finite amount of time.

7. A person does not really understand something until after teaching it to a computer Donald Knuth

8. The Role of the Algorithms in ComputING

9. Algorithms • Algorithm: Any well-defined computation procedure that takes some value, or set of values, as input and produces some value, or set of values, as output. • Or: tool for solving well specific computational problem. • Example: Sorting problem

10. Problem • Example: Sorting problem • Input: A sequence of n numbers • Output: A permutation of the input sequence such that . • Sorting algorithm: • The procedure that produces the sorted outputs for all possible inputs.

11. Properties • Aninstance of a problem consists of all inputs needed to compute a solution to the problem. • An algorithm is said to be correct if, for every input instance, it halts with the correct output. • A correct algorithm solves the given computational problem. An incorrect algorithm might not halt at all on some input instance, or it might halt with other than the desired answer.

12. What kind of problem can be solved by algorithm? • The Human Genome Project • The Internet Applications • Electronic Commerce with Public-key cryptography and digital signatures • Manufacturing and other commercial settings

13. Interesting problems • How to solve these problems • Shortest path (Ch. 24) • Longest common subsequence (Ch. 15) • Convex hull (CH. 33) • Characteristics of interesting problem • Find the best solutions from the candidate ones. • Practical application.

14. Hard Problems • Efficiency is speed, i.e., how long an algorithm takes to produce its result. • NP-Complete • no efficient algorithm for an NP-complete problem has ever been found, • NP-complete problems has the remarkable property that if an efficient algorithm exists for any one of them

15. Algorithm choice • There are a large number of good sorting algorithms. • Which algorithm is best for a given application • • It depends on: • The number of items to be sorted. • The extent to which the items are already sorted. • Possible restrictions on item values. • The kind of storage device to be used. • An algorithm is said to be correct if, for every input instance, it gives the correct output.

16. Algorithm efficiency • Algorithms devised to solve the same problem often differ in their efficiency. • These differences can be much more significant than differences due to hardware and software. Example: • Insertion sort algorithm takes time = c1n2 to sort n numbers, where c1 is a constant. • Merge sort algorithm takes time = c2 n lg n to sort n numbers, where c2 is a constant.

17. Algorithm efficiency • c1 < c2, but constants less significant in the running time than the input size. • lgn < n, but insertion sort is usually faster than merge sort for small input sizes. • Once the input size n becomes large enough, merge sort will become faster than insertion sort.

18. Algorithm efficiency • • Example: • Computer A executes one billion instruction per second (1 GHz) and executes insertion sort (c1n2). • Computer B executes one million instruction per second (1 MHz) and executes merge sort (c2 n lg n). • Assume c1=2 and c2=50. • Assume we need to sort 1 million numbers.

19. Algorithm efficiency • Computer A takes= 2*(106)2 instructions = 2000 s 109instructions/second • Computer B takes= 50*106* lg106 instructions = 100 s 107instructions/second

20. Algorithm efficiency • Imagine that we need to sort 10 million numbers; • Insertion sort takes 2.3 days. • Merge sort takes 20 minutes. • We analyze algorithms to improve them and choose the best one for a given problem.

21. Algorithm Questions • How to devise algorithms? • there are various design techniques. • example: divide and conquer. • How to analyze algorithms? • the task of determining how much computing time and storage and algorithm requires. • it needs mathematical skills. • best case, average case, and worst case.

22. Algorithm Questions • How to validate algorithms? • validation: showing that the algorithm computes the correct answer for all legal inputs. • after validation, verification process takes place.

23. Insertion sort • solves the sorting problem • Input: A sequence of n numbers • Output: A permutation of the input sequence such that .

24. Insertion sort

25. Insertion sort – pseudo code • InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} }

26. An Example: Insertion Sort i =  j =  key = A[j] =  A[j+1] =  30 10 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 269/17/2013

27. An Example: Insertion Sort i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10 30 10 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 279/17/2013

28. An Example: Insertion Sort i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30 30 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 289/17/2013

29. An Example: Insertion Sort i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30 30 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 299/17/2013

30. An Example: Insertion Sort i = 2 j = 0 key = 10A[j] =  A[j+1] = 30 30 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 309/17/2013

31. An Example: Insertion Sort i = 2 j = 0 key = 10A[j] =  A[j+1] = 30 30 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 319/17/2013

32. An Example: Insertion Sort i = 2 j = 0 key = 10A[j] =  A[j+1] = 10 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 329/17/2013

33. An Example: Insertion Sort i = 3 j = 0 key = 10A[j] =  A[j+1] = 10 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 339/17/2013

34. An Example: Insertion Sort i = 3 j = 0 key = 40A[j] =  A[j+1] = 10 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 349/17/2013

35. An Example: Insertion Sort i = 3 j = 0 key = 40A[j] =  A[j+1] = 10 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 359/17/2013

36. An Example: Insertion Sort i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 369/17/2013

37. An Example: Insertion Sort i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 379/17/2013

38. An Example: Insertion Sort i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 389/17/2013

39. An Example: Insertion Sort i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 399/17/2013

40. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 409/17/2013

41. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 419/17/2013

42. An Example: Insertion Sort i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 429/17/2013

43. An Example: Insertion Sort i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20 10 30 40 20 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 439/17/2013

44. An Example: Insertion Sort i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40 10 30 40 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 449/17/2013

45. An Example: Insertion Sort i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40 10 30 40 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 459/17/2013

46. An Example: Insertion Sort i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40 10 30 40 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 469/17/2013

47. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40 10 30 40 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 479/17/2013

48. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40 10 30 40 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 489/17/2013

49. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30 10 30 30 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 499/17/2013

50. An Example: Insertion Sort i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30 10 30 30 40 InsertionSort(A, n) {for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key} } 1 2 3 4 David Luebke 509/17/2013