CS214 – Data Structures Lecture 04&5: Quadratic Sorting

1. CS214 – Data StructuresLecture 04&5: Quadratic Sorting Slides by Dr-Mohamed El-Ramly, PhD Dr-Basheer Youssef

2. Agenda 0. Why Sorting? Quadratic Sorting algorithms Selection Sort Insertion Sort Bubble Sort Theoretical Boundaries of Sorting Problem STL support for sorting Sub-quadratic Sorting Merge Sort Shell Sort Quick Sort Heap Sort

3. Chapter Objectives • To learn how to use the standard sorting algorithms in STL • To learn how to implement the following sorting algorithms: selection sort, bubble sort, insertion sort, shell sort, merge sort, heap sort and quick. • To understand the difference in performance of these algorithms, and which to use for small, medium and large sets of data and for what types of data. Chapter 9: Sorting

4. Why Sorting? • The efficiency of data handling can be substantially increased if the data is sorted. • Imagine an unsorted phone directory or a dictionary! • Often, it is required to sort data before processing. • First, we need to decide on the sorting criterion • Second we need to decide on the sorting algorithm Chapter 9: Sorting

5. Which Sorting Algorithms? • To compare algorithm efficiency, we need a machine-independent criterion for evaluating their performance. • Two factors determine the algorithm performance: • The number of comparisons • The number of movements • We use Big-O notation for this purpose. Chapter 9: Sorting

6. What Affects Performance? • Initial Order of the Data • Does the algorithm recognize already ordered data? • How much time is spent in on ordering already ordered data? • Algorithm’s intelligence • Best / Worst / Average cases • Comparisons / Movements: simple or complex? • Do we compare int values or arrays? • Do we move int values or structures? Chapter 9: Sorting

7. Chapter 9: Sorting

8. The family of sorting methods Main sorting themes Address- -based sorting Comparison-based sorting count Sort Transposition sorting BubbleSort Divide and conquer Diminishing increment sorting Insert and keep sorted MergeSort QuickSort Insertion sort ShellSort

9. Insertion Sort • Based on the technique used by card players to arrange a hand of cards • Player keeps the cards that have been picked up so far in sorted order • When the player picks up a new card, he makes room for the new card and then inserts it in its proper place Chapter 9: Sorting

10. Insertion Sort Algorithm • For each array element from the second to the last (nextPos = 1) • Insert the element at nextPos where it belongs in the array, increasing the length of the sorted subarray by 1 Chapter 9: Sorting

11. next to be inserted sorted 3 4 7 12 14 33 14 20 21 38 10 55 9 23 28 16 temp less than 10 10 12 21 14 14 20 33 38 sorted 3 4 7 10 55 9 23 28 16 10 12 14 33 14 20 21 38 One step of insertion sort Chapter 9: Sorting

12. Psuedocode for Insertion Sort • insertionSort (data [], n) for (i = 1; i < n; i++) move all elements data[j]>data [i]by1position; place data [i]in its proper position; Chapter 9: Sorting

13. C++ Code for Insertion Sort • template <class T> • void insertionSort (T data[], int n) { for (int i = 1, j; i < n; i++) T tmp = data [i]; for (j = i; j > 0 && tmp < data [j-1]; j--) data [j] = data [j – 1]; data [j] = tmp; } Chapter 9: Sorting

14. Analysis of insertion sort • Insertion sort sorts array when really needed. • If the array is already sorted, moves are very few. • On the other hand, shifting can be time consuming. Chapter 9: Sorting

15. Best Case • Outer loop n – 1 • T tmp = data [i]; 1 move x n – 1 times • data [j] = tmp; 1 move x n – 1 times • tmp < data [j-1] 1 comparison x n – 1 • Moves: 2 (n - 1) • Comparisons: n – 1 • O (n) Chapter 9: Sorting

16. Worst Case • Happens when the data is in reverse order • Outer loop n – 1 • T tmp = data [i]; 1 move x n – 1 times • data [j] = tmp; 1 move x n – 1 times • Inner loop executes, 1, 2, …, n – 1 • For each inner loop, there is 1 comparison • For each inner loop, there is 1 move Chapter 9: Sorting

17. Worst Case • Comparisons: (1 + 2 + …. + (n - 1)) = n (n – 1) / 2  O(n2) • Moves: (1 + 2 + …. + (n - 1)) + 2 (n – 1) = n (n – 1) / 2 + 2 (n – 1) = n2/2 + 3n/2 – 2  O(n2) Chapter 9: Sorting

18. Average Case • This is an approximate analysis simpler than the book • Assume the data is randomly ordered • Outer loop n – 1 • T tmp = data [i]; 1 move x n – 1 times • data [j] = tmp; 1 move x n – 1 times • Inner loop executes, 1, 2, …, n – 1 • For each inner loop, there is on average i/2 comparison • For each inner loop, there is on average i/2 moves Chapter 9: Sorting

19. Average Case • Comparisons: (1 + 2 + …. + (n - 1))/2 = n (n – 1) / 4  O(n2) • Moves: (1 + 2 + …. + (n - 1))/2 + 2 (n – 1) = n (n – 1) / 4 + 2 (n – 1) = n2/4 + 7n/4 – 2  O(n2) Chapter 9: Sorting

20. Analysis of Insertion Sort • Maximum number of comparisons is O(n*n) • In the best case, number of comparisons is O(n) • The number of shifts performed during an insertion is one less than the number of comparisons or, when the new value is the smallest so far, the same as the number of comparisons • A shift in an insertion sort requires the movement of only one item whereas in a bubble or selection sort an exchange involves a temporary item and requires the movement of three items Chapter 9: Sorting

21. Chapter 9: Sorting

22. Selection Sort • Selection sort is a relatively easy to understand algorithm • Sorts an array by making several passes through the array, selecting the next smallest item in the array each time and placing it where it belongs in the array • It attempts to localize exchanges by putting the item directly in its final place. • Efficiency is O(n*n) Chapter 9: Sorting

23. SelectionSort • Given an array of length n, • Search elements0 through n-1 and select the smallest • Swap it with the element in location 0 • Search elements 1 through n-1 and select the smallest • Swap it with the element in location 1 • Search elements 2 through n-1 and select the smallest • Swap it with the element in location 2 • Search elements 3 through n-1 and select the smallest • Swap it with the element in location 3 • Continue in this fashion until there’s nothing left to search Chapter 9: Sorting

24. 2 2 7 2 2 4 2 4 7 4 5 8 5 8 8 7 5 5 8 5 7 8 4 4 7 Example and analysis of Selection Sort • The Selection Sort might swap an array element with itself--this is harmless, and not worth checking for • Analysis: • The outer loop executes n-1 times • The inner loop executes about n/2 times on average (from n to 2 times) • Work done in the inner loop is constant (swap two array elements) • Time required is roughly (n-1)*(n/2) • You should recognize this asO(n2) Chapter 9: Sorting

25. Psuedocode for Selection Sort • selectionSort (data [], n) for (i = 1; i < n-1; i++) select smallest element from data[i],…,data [n-1]; swap it with data [i]; Chapter 9: Sorting

26. C++ Code of Selection Sort • template <class T> • void selectionSort(T data[], int n) { for (int i = 0, j, least; i < n-1; i++) { for (j = i+1, least = i; j < n; j++) if (data [j] < data [least]) least = j; swap (data [least], data [i]); } } Chapter 9: Sorting

27. Analysis of Selection Sort • template <class T> • void selectionSort(T data[], int n) { for (int i = 0, j, least; i < n-1; i++) { for (j = i+1, least = i; j < n; j++) if (data [j] < data [least]) least = j; swap (data [least], data [i]); } } n - 1 n-1, n-2,…,1 1 comparison 1 swap Chapter 9: Sorting

28. Best / Worst / Average Case • It does not recognize order of data • Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2  O (n2) • Swaps: n – 1  O (n) • Moves: 3 (n – 1)  O (n) Chapter 9: Sorting

29. Analysis of Selection Sort • What conclusion can we make about Selection Sort? • How intelligent is Selection Sort? • What do you notice about the number of swaps? • Modify the algorithm to further reduce the number of swaps. Chapter 9: Sorting

30. Chapter 9: Sorting

31. Bubble Sort • Compares adjacent array elements and exchanges their values if they are out of order • Smaller values bubble up to the top of the array and larger values sink to the bottom Chapter 9: Sorting

32. 2 2 2 2 2 2 2 2 2 2 7 2 2 4 7 4 2 7 7 5 5 7 5 4 7 7 5 5 8 5 5 8 5 5 8 4 7 5 4 4 4 8 5 5 5 7 7 4 7 4 7 7 8 4 8 8 8 8 8 4 4 4 8 8 8 2 5 4 7 8 Example of Bubble Sort (done) Chapter 9: Sorting

33. Psuedocode for Bottom-up Bubble Sort • bubbleSort (data [], n) for (i = 1; i < n-1; i++) for (j = n - 1; j > i; --j) swap items j and j – 1 if out of order; Chapter 9: Sorting

34. C++ Code of Bottom-up Bubble Sort • template <class T> • void bubbleSort(T data[], int n) { for (int i = 0; i < n - 1; i++) { for (int j = n - 1; j > i; --j) if (data [j] < data [j-1]) swap (data [j], data [j - 1]); } Chapter 9: Sorting

35. Analysis of Bubble Sort • template <class T> • void bubbleSort(T data[], int n) { for (int i = 0; i < n - 1; i++) { for (int j = n - 1; j > i; --j) if (data [j] < data [j-1]) swap (data [j], data [j - 1]); } n - 1 n-1,…,1 1 comparison 1 swap Chapter 9: Sorting

36. Worst Case • Reverse ordered data • Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2  O (n2) • Swaps: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2  O (n2) • Moves: 3 n (n – 1) / 2  O (n2) Chapter 9: Sorting

37. Best Case • When array is ordered • Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2  O (n2)  Can it be improved to O(n) • Swaps: none  O (1) • Moves: none  O (1) Chapter 9: Sorting

38. Average Case • Comparisons: 1 + 2 + . . . + (n - 1) = n (n - 1) / 2  O (n2) • Swaps: 1/2 + 2/2 + . . . + (n - 1)/2 = n (n - 1) / 4  O (n2) • Moves: 3 n (n – 1) / 4  O (n2) Chapter 9: Sorting

39. Analysis of Bubble Sort • Provides excellent performance in some cases and very poor performances in other cases • Main problem: it moves data up, one step at a time • Works best when array is nearly sorted to begin with • Worst case number of comparisons is O(n2) • Worst case number of exchanges is O(n2) • Best case occurs when the array is already sorted • O(n) comparisons • O(1) exchanges Chapter 9: Sorting

40. Chapter 9: Sorting

41. Comparison of Quadratic Sorts • None of the algorithms are particularly good for large arrays Chapter 9: Sorting

42. Comparison of Quadratic Sorts • Bubble Sort, Selection Sort, and Insertion Sort are all O(n2) • As we will see later, we can do much better than this with somewhat more complicated sorting algorithms • WithinO(n2), • Bubble Sort is very slow, and should probably never be used for anything • Selection Sort is intermediate in speed • Insertion Sort is usually the fastest of the three--in fact, for small arrays (say, 10 or 15 elements), insertion sort is faster than more complicated sorting algorithms • Selection Sort and Insertion Sort are “good enough” for small arrays Chapter 9: Sorting

43. Theoretical Analysis of Sorting Problem • Quadratic algorithms are not efficient ? • Can we obtain better efficiency ? • We need a lower bound or best performance. • We focus on comparisons not moves because in ideal cases, you can avoid any moves (e.g., bubble sort) • What is the best estimate of the number of item comparisons if the array is randomly ordered? Chapter 9: Sorting

44. Chapter 9: Sorting

45. Chapter 9: Sorting

46. Chapter 9: Sorting

47. Theoretical Analysis of Sorting Problem • Every sorting algorithms can be represented by a decision tree. • n – elements array may have n! ways of ordering • But leaves may be more due to failures or repetitions Chapter 9: Sorting

48. Chapter 9: Sorting

49. Theoretical Analysis of Sorting Problem • An i-level complete decision tree has 2i-1leaves and 2i-1 -1 non-terminals and 2i-1 total nodes. • So, all non-complete trees with the same number of i levels have fewer nodes, i.e., k+m 2i-1 (m leaves, k nonterminals) and k  2i-1 -1 and m 2i-1 • n! are the possible ways of ordering an n-array • n!  m  2i-1 • log (n!) + 1  i (We assume i is the lowest level referring to max number of comparisons needed and longest path on the tree) Chapter 9: Sorting

50. Theoretical Analysis of Sorting Problem • i  log (n!) + 1 • The path length of a tree with n! nodes should be at least log (n!) • In other words O (log (n!)) is the best to expect in the worst case for the number of comparisons. • The same applies to average case. Chapter 9: Sorting