ICS 353 Design and Analysis of Algorithms Section 01 – 09:20-10:20am – 24:165

1. ICS 353Design and Analysis of AlgorithmsSection 01 – 09:20-10:20am – 24:165 Summer Semester 2005 - 2006 (053) King Fahd University of Petroleum & Minerals Information & Computer Science Department

2. Important Preliminaries • Instructor: Dr. Wasfi Al-Khatib وصفي الخطيب • Office: (22) 133-1 • Office hours: • SMW 08:10 – 09:10 am. • UT 10:20 – 11:20 am. • Phone: 1715 • email: wasfi@ccse.kfupm.edu.sa

3. Important Preliminaries (Cont.) • Prerequisite: ICS 202: Data Structures • Prerequisites by Topic: • Data structures including linked lists, arrays, stacks, queues, trees, and graphs, and • Knowledge of a high level structured language.

4. Text Book Introduction to Algorithms: Design Techniques and analysis By M. Alsuwaiyel.

5. Course Goals • To provide the students with the following: • The fundamentals of algorithms and algorithmic techniques, • The ability to decide on the suitability of a specific technique for a given problem, • The ability to analyze the complexity of a given algorithm, • The ability to design efficient algorithms for new situations, using as building blocks the techniques learned, and • Introducing the concept of NP-complete problems.

6. Course Contents • Basic Concepts in Algorithmic Analysis, Chapter1 • Mathematical preliminaries, Chapter 2 + • Review of Data Structures, Chapter 3 • Advanced Data Structures, Chapter 4 • Induction, Chapter 5 • Divide and Conquer, Chapter 6 • Dynamic Programming, Chapter 7 • Greedy Algorithms, Chapter 8 • Graph Traversal, Chapter 9 • NP- Complete Problems, Chapter 10

7. Grading Policy

8. Important Dates

9. Final Important Remarks • Homeworks: • Due at the beginning of class. • Late homeworks are not accepted. • Discussing homeworks with others (especially on WebCT) is highly encouraged. However, copying homeworks is NOT permitted and will be considered CHEATING. • Homeworks are crucial for learning and obtaining good grades. • Quizzes: 15 minute. Each covers material given since the last quiz or major exam. • Pop Quizzes: 5-10 minute. Each covers material given during the same lecture. • Exams HWs, & quizzes are generally CHALLENGING. • MAKE USE of my office hours.

10. Attendance • Attendance will be checked each class. • A bonus of 0.5 point (out of the maximum 100) will be given for each “additional” hour attended above the 38 hours • This shall come from the help sessions conducted for the two major exams and the final exam. • Unexcused Absences Policies: • The first three absences are FREE of charge. • The fourth absence is worth – 2 of your maximum 100 total. • Each subsequent absence, up to the seventh absence, is worth – 0.5. • The eighth absence will result in an automatic DN grade. • An unexcused absence can become an excused absence ONLY by an official letter from the Dean of Student’s office.

11. Your 24-Hour Right • One has 24 hours to object to the grade of a homework, [pop] quiz or a major exam starting from the end of the class time in which the graded exam papers have been distributed. • i.e. if you were not present in class during the distribution of exam papers, you loose this right. • If for some reason you cannot see me in person, within this period, send me an email requesting an appointment. The email, though, should be sent within the 24-hour time period.

12. Cairo Envelopes Examples: • Slept after Fajr • Checkpoints long lines • I got married / I had a divorce • I am graduating and the job is waiting for me How much can they increase your letter grade? NONE…………..even if you bring an elder!

13. So, What Can Increase my Grade? • Active participation • In Class • On WebCT • In my office hours • The bottom line is: Trying

14. Cheating Policies • Cheating in one category will result in a zero for the whole category. For example, if cheating is confirmed in homework 3, this will result in 0 out of 10% [which means that there is no need to submit any additional homework]. • An additional 5% will be deducted from the overall total (out of 100).

15. What is an algorithm? • An algorithm is defined as a finite set of steps, each of which may require one or more operations and if carried out on a set of inputs, will produce one or more outputs after a finite amount of time. • Examples of Algorithms • Examples of computations that are not algorithms

16. Properties of Algorithms • Definiteness: It must be clear what should be done. • Effectiveness: Each step must be such that it can, at least in principle, be carried out by a person using pencil and paper in a finite amount of time. E.g. integer arithmetic. • An algorithm produces one or more outputs and may have zero or more externally supplied inputs. • Finiteness: Algorithms should terminate after a finite number of operations.

17. Our Objective • Find the most efficient algorithm for solving a particular problem. • In order to achieve the objective, we need to determine: • How can we find such algorithm? • What does it mean to be an efficient algorithm? • How can one tell that it is more efficient than other algorithms?

18. In the First Chapter • We will answer the following two questions • What does it mean to be an efficient algorithm? • How can one tell that it is more efficient than other algorithms? based on some easy-to-understand searching and sorting algorithms that we may have seen earlier.

19. Searching Problem • Assume A is an array with n elements A[1], A[2], … A[n]. For a given element x, we must determine whether there is an index j; 1 ≤ j ≤ n, such that x = A[j] • Two algorithms, among others, address this problem • Linear Search • Binary Search

20. Linear Search Algorithm Algorithm: LINEARSEARCH Input: array A[1..n] of n elements and an element x. Output: j if x = A[j], 1 ≤ j ≤ n, and 0 otherwise. 1. j  1 2. while (j < n) and (x A[j]) 3. j j + 1 4. end while 5. if x = A[j] then return j else return 0

21. Analyzing Linear Search • One way to measure efficiency is to count how many statements get executed before the algorithm terminates • One should keep an eye, though, on statements that are executed “repeatedly”. • What will be the number of “element” comparisons if x • First appears in the first element of A • First appears in the middle element of A • First appears in the last element of A • Doesn’t appear in A.

22. Binary Search • We can do “better” than linear search if we knew that the elements of A are sorted, say in non-decreasing order. • The idea is that you can compare x to the middle element of A, say A[middle]. • If x < A[middle] then you know that x cannot be an element from A[middle+1], A[middle+2], …A[n]. Why? • If x > A[middle] then you know that x cannot be an element from A[1], A[2], …A[middle-1]. Why?

23. Binary Search Algorithm Algorithm: BINARYSEARCH Input: An array A[1..n] of n elements sorted in nondecreasing order and an element x. Output: j if x = A[j], 1 ≤ j ≤ n, and 0 otherwise. 1. low  1; high n; j  0 2. while (low ≤high) and (j = 0) 3. mid (low + high)/2 4. if x = A[mid] then j mid 5. else if x < A[mid] then high mid - 1 6. else low mid + 1 7. end while 8. return j

24. Worst Case Analysis of Binary Search • What to do: Find the maximum number of element comparisons • How to do: • The number of “element” comparisons is equal to the number of iterations of the while loop in steps 2-7. HOW? • How many elements of the input do we have in the • First iteration • Second iteration • Thrid iteration • … • ith iteration • The last iteration occurs when the size of input we have =

25. Theorem • The number of comparisons performed by Algorithm BINARYSEARCH on a sorted array of size n is at most

26. Merging Two Sorted Lists • Problem Description: Given two lists (arrays) that are sorted in non-decreasing order, we need to merge them into one list sorted in non-decreasing order. • Example: Input Output

27. Algorithm MERGE Algorithm: MERGE Input: An array A[1..m] of elements and three indices p, q and r, with 1 ≤ p ≤ q <r ≤ m, such that both the subarrays A[p..q] and A[q + 1..r] are sorted individually in nondecreasing order. Output: A[p..r] contains the result of merging the two subarrays A[p..q] and A[q + 1..r]. Comment: B[p..r] is an auxiliary array.

28. 1. s ← p; t ← q + 1; k ← p 2. while s ≤ q and t ≤ r 3. if A[s] ≤ A[t] then 4. B[k] ← A[s] 5. s ← s + 1 6. else 7. B[k] ←A[t] 8. t ← t + 1 9. end if 10. k ← k + 1 11. end while 12. if (s = q + 1) then B[k..r] ← A[t..r] 13. else B[k..r] ← A[s..q] 14. end if 15. A[p..r] ← B[p..r] Algorithm MERGE (Cont.)

29. Analyzing MERGE • Assuming arrays A[p,q] and A[q+1,r] • The least number of comparisons is which occurs when • The most number of comparisons is which occurs when • The number of element assignments performed is

30. Selection Sort Algorithm: SELECTIONSORT Input: An array A[1..n] of n elements. Output: A[1..n] sorted in nondecreasing order. 1. for i  1 to n - 1 2. k i 3. for j i + 1 to n {Find the index of the ithsmallest element} 4. if A[j] < A[k] then k j 5. end for 6. if k i then interchange A[i] and A[k] 7. end for

31. 4 2 8 5 9 Selection Sort Example

32. 4 9 9 4 5 4 5 4 2 4 8 8 8 8 8 2 2 2 5 2 9 9 9 5 5 Selection Sort Example i k 1 2 2 5 3 5 4 4

33. Analyzing Selection Sort • We need to find the number of comparisons carried out in line #4: • For each iteration of the outer for loop, how many times is line #4 executed? • Therefore, in total, line #4 is executed • The number of element Interchanges (swaps): • Minimum: • Maximum: • NOTE: The number of element assignments is 3 times the number of element interchanges

34. Insertion Sort Algorithm: INSERTIONSORT Input: An array A[1..n] of n elements. Output: A[1..n] sorted in nondecreasing order. 1. for i  2 to n 2. x A[i] 3. j i - 1 4. while (j > 0) and (A[j] > x) 5. A[j + 1] A[j] 6. j j - 1 7. end while 8. A[j + 1] x 9. end for

35. Insertion Sort Example 5 2 9 8 4

36. Insertion Sort Example x=2 5 2 9 8 4 x=9 2 5 9 8 4 x=8 2 5 9 8 4 x=4 2 5 8 9 4 2 4 5 8 9

37. Analyzing Insertion Sort • The minimum number of element comparisons is which occurs when • The maximum number of element comparisons is which occurs when • The number of element assignments is

38. Bottom-Up Merge Sort • Informally, the algorithm does the following • 1. Divide the array into pairs of elements (with possibly single elements in case the number of elements is ) • 2. Merge each pair in non-decreasing order (with possibly a single “pair” left) • 3. Repeat step 2 until there is only one “pair” left.

39. 4 2 3 1 8 10 7 5 9 12 6 Bottom-Up Merge Sort Example

40. 5 5 5 4 5 5 2 2 2 2 2 2 9 9 9 3 9 9 8 8 8 8 1 8 4 4 4 4 4 8 12 10 12 12 12 12 7 7 7 7 7 7 1 5 1 1 1 1 3 3 3 9 3 3 6 6 6 6 6 10 10 10 10 10 12 6 Bottom-Up Merge Sort Example

41. Algorithm BOTTOMUPSORT Algorithm: BOTTOMUPSORT Input: An array A[1..n] of n elements. Output: A[1..n] sorted in nondecreasing order. 1. t ← 1 2. while t < n 3. s ←t; t ← 2s; i ← 0 4. while i + t ≤n 5. MERGE(A, i + 1, i + s, i + t) 6. i ←i + t 7. end while 8. if i + s < n then 9. MERGE(A, i + 1, i+ s, n) 10. end while

42. Analyzing Algorithm BOTTOMUPSORT • With no loss of generality, assume that the size of the array, n, is a power of 2. • In the first iteration, we have pairs that are merged using element comparisons. • In the second iteration, we have pairs that are merged using • …. • In the jth iteration, we have pairs that are merged using • The outer while loop is executed times. • Therefore,

43. Analyzing Algorithm BOTTOMUPSORT • What about the number of element assignments?

44. Time Complexity • One way of measuring the performance of an algorithm is how fast it executes. The question is how to measure this “time”? • Is having a digital stop watch suitable?

45. Order of Growth • As measuring time is subjective to many factors, we look for a more “objective” measure, i.e. the number of operations • Since counting the exact number of operations is cumbersome, sometimes impossible, we can always focus our attention to asymptotic analysis, where constants and lower-order terms are ignored. • E.g. n3, 1000n3, and 10n3+10000n2+5n-1 are all “the same” • The reason we can do this is that we are always interested in comparing different algorithms for arbitrary large number of inputs.

46. Example Growth rate for some function

47. Example Growth rate for same previous functions showing larger input sizes

48. Running Times for Different Sizes of Inputs of Different Functions

49. Asymptotic Analysis: Big-oh (O()) • Definition: For T(n) a non-negatively valued function, T(n) is in the set O(f(n)) if there exist two positive constants c and n0 such that T(n) cf(n) for all n > n0. • Usage: The algorithm is in O(n2) in [best, average, worst] case. • Meaning: For all data sets big enough (i.e., n>n0), the algorithm always executes in less than or equal to cf(n) steps in [best, average, worst] case.

50. Big O() • O() notation indicates an upper bound. • Usually, we look for the tightest upper bound: • while T(n) = 3n2 is in O(n3), we prefer O(n2).