Design and Analysis of Algorithms

1. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Design and Analysis of Algorithms Faculty Name : Shamiel Hashim Ebrahim Email: s.bakheit@uoh.edu.sa Office Address: B11 A Room: 203 Office Hours: See Table Course web site : ics353Material.wikispaces.com

2. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Text Book: M.H. Alsuwaiyel, Algorithms Design Techniques and Analysis, Publishing House of Electronics Industry, 2003. Anany Levitin, Introduction to the Design and Reference Books: Analysis of Algorithms, Tsinghua University Press, 2003. Thomas H. Cormen etc., Introduction to Algorithms (Second Edition), Higher Education Press & The MIT Press. Others, search on the website with keyword �Design and Analysis of algorithm�, you will find more... Text Book and Reference Books

3. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Grading Schemes

4. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Please finish your assignments and projects independently. If you really meet problems, you can seek help from me or other students, but remember not copying, CHEATING! Please sign attendance when needed, but remember not signing for others, CHEATING!at the beginning of the lcecture . Attendence will taken at the beginning of lecture Power off (or ring off) your mobile phone Feedback (positive or negative) is encouraged and welcome, and IMPORTANT. (ask questions!) Thinking, before, in and after class is encouraged and IMPORTANT. Caution

5. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Data Structure Discrete Mathematics C, Java or other programming languages Advanced Mathematics Course Prerequisite

6. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Donald E. Knuth stated �Computer Science is the study of algorithms� Cornerstone of computer science. Programs will not exist without algorithms. Closely related to our lives Help to guide how others analyze and solve problems Help to develop the ability of analyzing and solving problems via computers Very interesting if you can concentrate on this course

7. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Behind every successful programming project there is a good design. If the underlying design is bad, then often no amount of fine-tuning is going to make a substantial difference. Usually design of any algorithm is chosen from existing techniques that are discussed in data structure course and based upon the selection proper design steps are formed. �To be sure that your design is the most suitable design for the underlying problem, you must develop the ability of analyzing the design you made. Why Study this Course? cont

8. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Syllabus (sequence of modules )

9. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A The word Algorithm comes from the name of the Muslim author Abu Ja�far Mohammad ibn Musaal-Khowarizmi one of the most influential mathematicians in 9th century in Baghdad, wrote a textbook in Arabic about adding, multiplying, dividing numbers, and extracting square roots and computing p. Much of al-Khwarizmi�s work was written in a book titled al Kitab al-mukhatasar fi hisab al-jabrwa�l-muqabalah (The Compendious Book on Calculation by Completion and Balancing). It is from the titles of these writings and his name that the words algebra and algorithm are derived.�. Algorithm: A brief History

10. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Notion: Algorithms An algorithm is a sequence of unambiguous, well-defined instructions for solving a computational problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time. An algorithm is thus a procedure that consist of a finite set of instructions which given an input enables us to obtain an output .

11. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Notion: Algorithms More precisely, an algorithm is a method or process to solve a problem satisfying the following properties: Finiteness terminates after a finite number of steps Definiteness Each step must be rigorously and unambiguously specified. Input Valid inputs must be clearly specified. Output can be proved to produce the correct output given a valid can be proved to produce the correct output given a valid input. Effectiveness Steps must be sufficiently simple and basic. -e.g., check if 2 is the largest integer n for which there is a solution to the equation xn + yn = zn in positive integers x, y, and z

12. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Model of Computation Any algorithm can be analyzed when we study it as a pure mathematical entity, and so ignore issues such as programming language, machine, and operating system. Our analysis be as independent as possible of the variations in machine, operating system, compiler, or programming language. Unlike programs, algorithms to be understood primarily by people (i.e. programmers) and not machines. Thus gives us quite a bit of flexibility in how we present our algorithms, and many low-level details may be omitted .

13. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A A Simple Example Below is an algorithm SIMPLE in form of pseudo code SIMPLE 1 for x ? 4 to 1 2 do x * 2 3 x? x- 1 4 print x For analysis we will execute the code and calculate how many times each line of our algorithm is executed.

14. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A First hand Analysis SIMPLE 1 for x ? 4 to 1 2 do x * 2 3 x? x- 1 4 print x

15. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A First hand Analysis So we have total of 12 executions If one operation takes 10 nanoseconds to execute then total execution time for our SIMPLE algorithm will be 10 * 12= 120 nanoseconds. This is the running time of our algorithm.

16. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Examples of Algorithms � Computing the Greatest Common Divisor of Two Integers gcd(m, n): the largest integer that divides both m and n. First try -- Euclid�s algorithm: gcd(m, n) = gcd(n, m mod n) Step1: If n = 0, return the value of m as the answer and stop; otherwise, proceed to Step 2. Step2: Divide m by n and assign the value of the remainder to r. Step 3: Assign the value of n to m and the value of r to n. Go to Step 1.

17. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Lecture 2 Chapter 1- Basic Concepts in Algorithmic Analysis Section:1.3

18. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Analysis of Classical Algorithms In this lecture tow simple problems are reviewed ,the searching problem and the problem of merging two sorted lists . The objectives of this lecture:- To review and analyze two simple search algorithms which is :linear search and Binary search . To introduce and analysis the problem of merging two sorted lists

19. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A The Search Problem 1- THE SEARCH PROBLEM : Given an element x and a list of n elements stored in an array A of size n , the problem is that of finding an index j ,1=j =n ,such that x=A[j] if x is in A and j=0 otherwise. We will review tow method of searching :- 1- linear (Sequential ) search . 2- Binary Search.

20. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1.1-The Linear Search Algorithm Algorithm description :- Given as input an array A[1..n] and an element x, scan the entries in A one by one starting from the first element and compare each entry with xif after j comparisons , 1=j =n the search is successful i.e. x=A[j] , j is returned otherwise 0 is returned which mean the search is unsuccessful It called the linear search as the number of comparisons grows linearly with the size of the sequence. it also referred to as sequential search

21. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1-The Linear Search Algorithm The formal Description of the linear search Algorithm is :- Input : An array A[1..n] of n elements and an element x . Output : j if x=A[j] ,1=j =n and 0 otherwise 1. j?1 2. While (j<n) and x?A[j] 2.1 j ?j+1 2.2 End while 3.If x=A[j] then return j else return 0

22. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1-The Linear Search Algorithm In the algorithm element comparisons are done in the while statement (Step 2) of the algorithm (2. while j<n) and x?A[j] The minimum number of comparisons occurs when the element that we looking for is the first element in the array .in this case there only one element comparison The maximum number of comparisons occurs when the element that we looking for (x)is not among the elements of the array.the result is n comparison . So the number of element comparison in the linear search algorithm is between (1 and n ) .

23. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1.2-The Binary Search Algorithm As we see in the above algorithms there no information about the ordering of the elements in A is given ,if we given that the element in A is sorted in nondecreasing order then there more efficient algorithm such as the binary search which describe as follow :- Let A[low .. high] be a nonempty array of elements sorted in nondecreasing order, Let A[mid] be the middle element and suppose that x>A[mid] . we observe that if x is in A , then it must be one of the elements A[mid +1]�..A[high] so we only need to search of x in this rang and the elements in A[low�mid-1] are discarded in subsequent comparisons. WHY ?

24. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1.2-The Binary Search Algorithm cont. Also if x<A[mid] we only need to search in A[low�mid-1]This results in an efficient strategy which, because of its repetitive halving is referred to as Binary Search

25. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A 1.2-The Binary Search Algorithm cont. The formal Description of the linear search Algorithm is :- Input : An array A[1..n] of n elements sorted in nondecreasing order and an element x . Output : j if x =A[j], 1=j =n and 0 otherwise 1- low ?1; high ?n; j ?0 2- while (low=high) and (j=0) 3-mid ? 4- if x =A[mid] then j ? mid 5- else if x<A[mid] then high ?mid-1 6- else low ?mid+1 7- end while 8- return j

26. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Analysis of the Binary search algorithm :- In our analysis we assume that each three-way �comparison (if �then-else) count as one comparison . The minimum number of element comparisons=1 which obtained when the element we are looking for is in the middle of the array To compute the maximum number of element comparisons performed by Algorithm BinarySearch we assume that x is greater than or equal to all elements in the array then the remaining elements in A[1..n]in the second iteration will be as n is even or odd Similarly in the third iteration it will be

27. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Analysis of Binary Search algorithm In general the jth pass through the while loop of the algorithm , the number of the remaining elements is reduced to [n/2j-1] So the iterations continue until : -Either x is found or the size of the subsequence reaches 1 This means we look for j such that floor n/2j-1 =1 By the definition of the floor function and we can found that:- 1=n/2j-1<2 Or 2j-1 =n =2j (by multiplying by 2j-1) Or j-1 =log2 n<j (by taking log of the three terms ) Since j is integer we conclude that j= log n +1 as the maximum number of comparisons in the Binary search algorithm

28. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Binary Search Algorithm-Example Suppose that we want to search for x=35 in A[1..14]= The execution of the binary search algorithm on the above array is as follow :- First : the index (referred as mid in algorithm Binary Search ) of the array element to be compared with the value x is computed using the equation : Mid =[(low+high)/2]= [(1+14)/2]=7 And the value x=35 is compared with A[7]=10 and 35>A[7] the portion of the arrayA[1..7] is discard. Thus the remaining portion of the array to be searched next is the subarray :-

29. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Second : repeat the process of the computing the index of the array among the remaining subarray A[8..14] to be compared with x. In this case mid=[(low+high)/2]=[(8+14)/2]=11 Then x is compared with A[11]=23 and we found that 35>A[11] ,so the portion of the array A[8..11] is discarded . Now the search reduced to the following subarray : A[12..14]= Third :Then we repeat the process of computing the mid which is Mid=[(low+high)/2]=[12+14]/2=13 Then compared with A[13]=32 and since 35>A[13] then the portion of the array A[12..13] is discarded and Now the search is limited in one element array A[14]= Finally : x compared with A[14]=35 As mid=[14+14)/2]=14 and we found that x=35=A[14] And the search is successfully completed

30. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A The Binary Search Algorithm and the Binary Search Decision tree It is possible to describe the binary search algorithm using the binary search decision tree . Definition :it is a binary tree that exhibits the behavior of the binary search algorithm in pictorial representation , in a binary decision tree a series of decisions leads to solution . In terms of te binary search problem it is binary tree used to locate items based on a sequence of comparisions where each comparison tells us whether we have: Locate the item or Follow the left subtree or follow the right subtree

31. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Binary Search Decision Tree Following figure shows the decision tree corresponding to the array given in examples 1.1:-

32. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Binary search Decision Tree -Notes 1- the decision tree is a function of the number of elements in the array only . 2-the maximum number of comparisons is one plus the height of the corresponding decision tree . 3- the height of such tree is log n.

33. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Theorem :- The number of comparisons performed by algorithm Binary search on a sorted array of size n is at most log n +1 Binary search Decision Tree -Theorem

34. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Lecture 3 Merge Sort Algorithm

35. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merging two sorted list Problem Statement:- Suppose we have an array A[1..m] of m elements and p ,q and r with 1=p=q<r<m, such that both subarrays A[p...q] and A[q+1..r] are individually in nondecreasing order we want to rearrange the elements A[p..q] and A[q+1..r] to get the array A[p..r] sorted in nondecreasing order this process is referred as merging A[p..q] with A[q+1..r] . An algorithm of merging two array works as follows :- * maintain two pointers s and t that points to A[p] and A[q+1] respectively .

36. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merging two sorted list * prepare an empty array B[p..r] as temporary storage, to store the intermediate result of the merge operation. * each time we compare elements A[s] and A[t] and append the smaller of the two to the auxiliary array B . * If they are equal we will choose to append A[s]. * Then we update the pointers : if A[s] =A[t] then we incensement s otherwise we increment t. This process ends when when s=q+1 or t=r+1 . So in the first case (s=q+1) we append the remaining elements A[t..r] to B. OR in the second case (t=r+1) we append A[s..q] to B. *Finally the array B[p..r] is copied back to A[p..r].

37. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merge Algorithm Input : An array A[1..m] of m elements and p ,q and r with 1=p=q<r<m, such that both subarrays A[p...q] and A[q+1..r] are individually in nondecreasing order. Output : A[p..q] contains the result of merging the two subarray A[p..q] and A[q+1..r] .

38. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Comment :B[p..r] is an auxiliary array . s?p ; t ?q+1; k ?p . While s =q and t = r. If A[s] =A[t] then B[k] ?A[s] s ?s+1 Else B[k] ?A[t] t?t+1

39. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A End if k ?k+1 End while If s=q+1 then B[k..r] ?A[t..r] Else B[k..r] ?A[s..q] End if A[p..r] ?B[p..r]

40. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merge Algorithm-Example Consider the two subarray A[p..q] and A[q+1..r] below need to e merged Input : A[p..q] and A[q+1..r] Where p =1 ,q=3 and q+1=4 ,r=9 The pointers s and t which points to A[p] and A[q+1]

41. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Compare A[s] with A[t] to append the smaller of the two to the auxiliary array B , If the two elements are equal we will choose to append A[s] as follow :- First comparison : since A[s]=A[t] (4<12) then shift A[s] to B[K] B[k..r]= Then s=s+1and k=k+1

42. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Second comparison : A[s] = A[t] (7<12) then shift A[s+1] to B[k+1] B[k..r]= Then s=s+1 and k=k+1 Third comparison: since A[s] = A[t] (9<12) then shift A[s+2] to B[k+2] B[k..r]=

43. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Now the process of comparison end when either s=q+1 or t=r+1 In this case s=q+1 so the comparison ends Next move all the remaining elements in array A to array B to get : Array B Finally the elements in array B moved back to get the final result in array A.

44. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Number of comparison The previous example require 3 element comparisons .which is least number of comparison which happened when each entry in the smaller subarray is less than all entries in the larger subarray .

45. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merge two sorted lists - observations 1- the number of element comparisons performed by merge algorithm to merge two nonempty arrays of size n1 and n2 where n1=n2 into a sorted array of size n=n1+n2 is between n1 and n-1. if two array size are n/2 and n/2 the number of comparisons needs is between n/2 and n-1 . 2- the number of elements assignments performed by Merge Algorithm to merge two arrays into one sorted array of size n is exactly 2n .

46. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Merge sort summary Divide: divide the n-element sequence into two subproblems of n/2 elements each. Conquer: sort the two subsequences recursively using merge sort. If the length of a sequence is 1, do nothing since it is already in order. Combine: merge the two sorted subsequences to produce the sorted answer.

47. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A Assignment No1 Apply merge algorithm for the following subarray get the number of comparison requires A[p..q] = And A[q+1 ..r]=

48. UOH-CCSE-ICS353-prepared by:Shamiel .H -Natalia .A ICS 353 Lectures Notes - shamiel Hashim Ebrahim Reference Book : Algorithms Design Techniques and Analysis by : M.H.Alsuwaiyel