Sorting

1. Sorting • We have actually seen already two efficient ways to sort:

2. A kind of “insertion” sort • Insert the elements into a red-black tree one by one • Traverse the tree in in-order and collect the keys • Takes O(nlog(n)) time

3. Heapsort (Willians, Floyd, 1964) • Put the elements in an array • Make the array into a heap • Do a deletemin and put the deleted element at the last position of the array

4. Quicksort (Hoare 1961)

5. quicksort Input: an array A[p, r] Quicksort (A, p, r) if (p < r) then q = Partition (A, p, r) //q is the position of the pivot element Quicksort(A, p, q-1) Quicksort(A, q+1, r)

6. p r j i 2 8 7 1 3 5 6 4 j i 2 8 7 1 3 5 6 4 j i 2 8 7 1 3 5 6 4 j i 2 1 7 8 3 5 6 4 i j 2 8 7 1 3 5 6 4

7. j i 2 1 7 8 3 5 6 4 j i 2 1 3 8 7 5 6 4 j i 2 1 3 8 7 5 6 4 j i 2 1 3 8 7 5 6 4 j i 2 1 3 4 7 5 6 8

8. 2 8 7 1 3 5 6 4 r p Partition(A, p, r) x ←A[r] i ← p-1 for j ← p to r-1 do if A[j] ≤ x then i ← i+1 exchange A[i] ↔ A[j] exchange A[i+1] ↔A[r] return i+1

9. Analysis • Running time is proportional to the number of comparisons • Each pair is compared at most once  O(n2) • In fact for each n there is an input of size n on which quicksort takes Ω(n2) time

10. But • Assume that the split is even in each iteration

11. T(n) = 2T(n/2) + n How do we solve linear recurrences like this ? (read Chapter 4)

12. Recurrence tree n T(n/2) T(n/2)

13. Recurrence tree n n/2 n/2 T(n/4) T(n/4) T(n/4) T(n/4)

14. Recurrence tree n n/2 n/2 logn T(n/4) T(n/4) T(n/4) T(n/4) In every level we do bn comparisons So the total number of comparisons is O(nlogn)

15. Observations • We can’t guarantee good splits • But intuitively on random inputs we will get good splits

16. Randomized quicksort • Use randomized-partition rather than partition Randomized-partition (A, p, r) i ← random(p,r) exchange A[r] ↔ A[i] return partition(A,p,r)

17. On the same input we will get a different running time in each run ! • Look at the average for one particular input of all these running times

18. Expected # of comparisons Let X be the expected # of comparisons This is a random variable Want to know E(X)

19. Expected # of comparisons Let z1,z2,.....,zn the elements in sorted order Let Xij = 1 if zi is compared to zj and 0 otherwise So,

20. by linearity of expectation

21. Consider zi,zi+1,.......,zj ≡ Zij Claim: zi and zj are compared  either zi or zj is the first chosen in Zij Proof: 3 cases: • {zi, …, zj} Compared on this partition, and never again. • {zi, …, zj} the same • {zi, …, zk, …, zj} Not compared on this partition. Partition separates them, so no future partition uses both.

22. just explained = Pr{zi or zj is first pivot chosen from Zij} = Pr{zi is first pivot chosen from Zij} + Pr{zj is first pivot chosen from Zij} mutually exclusive possibilities Pr{zi is compared to zj} = 1/(j-i+1) + 1/(j-i+1) = 2/(j-i+1)

23. Simplify with a change of variable, k=j-i+1. Simplify and overestimate, by adding terms.

24. Lower bound for sorting in the comparison model

25. A lower bound • Comparison model: We assume that the operation from which we deduce order among keys are comparisons • Then we prove that we need Ω(nlogn) comparisons on the worst case

26. Model the algorithm as a decision tree

27. Insertion sort 1:2 z x x y y z x y x y x y z z y z x x z y y x z y z z z z x y y x x > < 2:3 2:3 < > > 1:2 1:2 > > < <

28. Quicksort 1:3 x x x x y z y z z x y y x z x z y y z x z z y y z y x > < 2:3 2:3 < > < > 2:3 1:2 > < > <

29. Important observations • Every algorithm can be represented as a (binary) tree like this • For every node v there is an input on which the algorithm reaches v • The # of leaves is n!

30. Important observations • Each path corresponds to a run on some input • The worst case # of comparisons corresponds to the longest path

31. The lower bound • Let d be the length of the longest path n! ≤ #leaves ≤ 2d log2(n!) ≤ d

32. Lower bound for sorting • Any sorting algorithm based on comparisons between elements requires (n log n) comparisons.

33. Beating the lower bound • We can beat the lower bound if we can deduce order relations between keys not by comparisons Examples: • Count sort • Radix sort

34. Count sort • Assume that keys are integers between 0 and k A 2 3 0 5 3 5 0 2 0

35. Count sort • Allocate a temporary array of size k: cell x counts the # of keys =x A 2 3 0 5 3 5 0 2 5 C 0 0 0 0 0 0

36. Count sort A 2 3 0 5 3 5 0 2 5 C 0 0 1 0 0 0

37. Count sort A 2 3 0 5 3 5 0 2 5 C 0 0 1 1 0 0

38. Count sort A 2 3 0 5 3 5 0 2 5 C 1 0 1 1 0 0

39. Count sort A 2 3 0 5 3 5 0 2 5 C 2 0 2 2 0 3

40. Count sort • Compute prefix sums of C: cell x holds the # of keys ≤ x (rather than =x) A 2 3 0 5 3 5 0 2 5 C 2 0 2 2 0 3

41. Count sort • Compute prefix sums of C: cell x holds the # of keys ≤ x (rather than =x) A 2 3 0 5 3 5 0 2 5 C 2 2 4 6 6 9

42. Count sort • Move items to output array A 2 3 0 5 3 5 0 2 5 C 2 2 4 6 6 9 B / / / / / / / / /

43. Count sort A 2 3 0 5 3 5 0 2 5 C 2 2 4 6 6 9 B / / / / / / / / /

44. Count sort A 2 3 0 5 3 5 0 2 5 C 2 2 4 6 6 8 B / / / / / / / / 5

45. Count sort A 2 3 0 5 3 5 0 2 5 C 2 2 3 6 6 8 B / / / 2 / / / / 5

46. Count sort A 2 3 0 5 3 5 0 2 5 C 1 2 3 6 6 8 B / 0 / 2 / / / / 5

47. Count sort A 2 3 0 5 3 5 0 2 5 C 1 2 3 6 6 7 B / 0 / 2 / / / 5 5

48. Count sort A 2 3 0 5 3 5 0 2 5 C 1 2 3 5 6 7 B / 0 / 2 / 3 / 5 5

49. Count sort A 2 3 0 5 3 5 0 2 5 C 0 2 2 4 6 6 B 0 0 2 2 3 3 5 5 5

50. Count sort • Complexity: O(n+k) • The sort is stable • Note that count sort does not perform any comparison