CS305/503, Spring 2009 Amortized Analysis

1. CS305/503, Spring 2009Amortized Analysis Michael Barnathan

2. Here’s what we’ll be learning: • Theory: • Amortized Analysis • Java: • TreeSet • HashSet

3. Traditional Asymptotic Analysis • Looks at the behavior of one operation. • One insertion takes O(n) time… • One search takes O(log n) time… • One, one, one. • If every operation takes the same amount of time, this is perfectly fine. We can figure out the cost of the sequence. • What is the total complexity of n operations, each taking time proportional to O(n)? • However, this is not always the case. • What about Vectors, which increase in size when they are filled? • Each insertion at the end takes O(1) time, until the array is full, upon which the next insertion takes O(n) time.

4. Vector Doubling in Traditional Analysis • Suppose we perform n insertions on a vector that employs the doubling strategy. • In traditional analysis, every operation has the same cost. So what is the worst case cost of insertion into an array? • O(n), because in the worst case, we double. • This is despite the fact that most insertions take O(1) time, because the majority do not double. • We perform n insertions, each taking O(n) time. • What is the bound upon the complexity? • n * O(n) = O(n2). • This is clearly not a tight bound.

5. Amortization • Amortized analysis analyzes worst-case performance over a sequence of operations. • It is not an average case analysis; it is an “average of worst cases”. • Going back to Vector insertion: • If we perform 6 insertions into a Vector of size 5, 5 of those insertions will take 1 unit of time. The sixth will take 6 units. (Both in the worst case) • Since all 6 insertions will take 11 units of time, one insertion contributes roughly 2 time units, not on average, but in the worst case. • 2 is a constant. We would expect constant time behavior on insert. • An individual insertion may take longer (the sixth insertion takes 6 units of time, for example), but it will make up for it by preparing subsequent insertions to run quickly (by doubling the array).

6. Methods of Amortization • There are three commonly employed amortized analysis techniques. • From least to most formal: • The Aggregate Method: • Count the number of time units across a sequence of operations and divide by the number of operations. • The Accounting Method: • Each operation “deposits” time, which is then used to “pay for” expensive operations. • The Potential Method: • A “potential function” φ is defined based on the change in state brought about by each operation and the difference in potential is added to the total cost (this difference may be negative). • We won’t go into too much detail on this method. • Each method has its limitations.

7. The Aggregate Method • This is the simplest method of analysis. • Simply add the worst-case cost of each operation in a sequence up, then divide by the total number of operations in the sequence: • The cost of each operation is very often defined asymptotically, not as a number. • But that’s OK; O(n) means “a linear function of n”. • So O(n) / n = O(1), O(n2) / n = O(n), and so forth.

8. The Aggregate Method – Example: So far, the amortized cost is [n * O(1)] / n = O(1). The fourth insertion doubles the array, an O(n) operation. Now the amortized cost is [(n-1) * O(1) + O(n)] / n = O(1) + O(1) = O(1).

9. Caveats • The lack of formalism in the aggregate method has some consequences. • Specifically, when using the aggregate method, be careful with your asymptotics! • O(n) at the 4th insertion is very different from O(n) at the 32768th insertion! • It is thus sometimes useful to define the elementary cost of inserting without doubling as simply “1” and to use exact numbers.

10. Again, with numbers. So far, the amortized cost is [3 * 1] / 3 = 1. The fourth insertion doubles the array, an O(n) operation. Now the amortized cost is [6 * 1 + 4] / 7 = 10 / 7 = 1.43.

11. How it Converges • It turns out that this is always constant-time, no matter how many insertions you do: • For a sequence of n operations, the algorithm will double at n, n/2, n/4, n/8, … • The number of elements in the array at each double is the cost of that doubling step (because it’s O(n)). • So the cost is defined by a convergent series: • So, at worst, it will take you thrice as long to use a doubling array as a preallocated one. • 3 * O(1) = O(1); this is still constant-time. Cost of insertion Cost of doubling

12. The Accounting Method • The accounting method begins by assigning each elementary operation a cost of $1. • The cost your analysis returns, of course, is then in terms of how long those operations take. • Each operation will then pay for: • The actual cost of the operation, and • The future cost of keeping that element maintained (for example, copying it in the array). • We save in advance so we have something to “spend” when we double. • We call the saved money “the bank”. • The bank balance never goes negative; there are no subprime loans. • This is sort of difficult because it requires us looking ahead to see what happens when we double.

13. What’s the cost? • Each element we insert costs $1 immediately. • When doubling: • We will have to move each element to a new array; each move costs $1. • We will have to create a new element for each existing element (because we’re doubling the size). Something is eventually going to fill this spot as well. This will cost $1. • So the total cost is $3 per insertion. • $1 for now, $2 for the future. • This is the same answer we received using the aggregate method. • But requires careful inspection to arrive at.

14. Does it work? • Remember, the bank must never go negative. • Doubling costs $n+1: $n to copy the n elements, $1 for the insertion that immediately follows. • Each insertion pays $3 and costs $1, so $2 goes into the bank at each non-doubling step. • And each doubling costs $n+1.

15. Yes, it does. Red fields represent insertions that cause the array to double.

16. Potential Method • Instead of “saving” and paying the cost later, the potential method measures the “potential difference” between two adjacent operations. • This is defined by a potential function φ. • Φ(0) = 0. • Φ(i) ≥ 0 for all i. • The amortized cost of operation i is determined by the actual cost plus the difference in potential: • aci = ci + [φ(i) - φ(i-1)] • The total cost is the sum of these individual costs: “Telescoping”: all phi terms except n and 0 cancel; they’re both added and subtracted.

17. Potential Method Example • For the array doubling problem, • ci = 1 if i is not a power of 2, i otherwise. (Direct cost of inserting item i) • How did we get this? The first term counts the cost we’ve accumulated for creating each of the first i elements. The second term subtracts the costs we’ve already paid in previous doublings (it represents the current array capacity). • If i-1 is a power of 2, • ci = i (we’re going to double) • φ(i) = 2i - 2(i -1)= 2 • φ(i-1) = 2(i-1) - (i-1) = i - 1 • aci = ci + φ(i) - φ(i-1) = i+ 2 - (i - 1) = i+ 2 – i + 1 = 1 + 2 = 3. • If i-1 is not a power of 2, • ceil(lg(i)) = ceil(lg(i-1)), so the potential terms cancel. • ci = 1 (no doubling) • φ(i) = 2i • φ(i-1) = 2(i-1) = 2i - 2 • aci = ci + φ(i) - φ(i-1) = 1+ 2i - (2i - 2) = 1+ 2i - 2i + 2 = 1 + 2 = 3. • So we get the same answer as in the other methods.

18. Careful • The potential method appears the most rigorous of the three, but you still need to deduce the potential function from the problem. • The other two methods are less prone to mistakes.

19. Sets • A set is a data structure that can be used to store records in which the key is the same as the value, keeping all elements in the set unique. • There are two types in Java: • TreeSets: Sorted Unique Containers. • HashSets: Unsorted Unique Containers. • A multiset, or bag, is like a set, but without the uniqueness constraint.

20. Special Properties • Elements in a set class are guaranteed unique. • Attempting to insert an element that already exists will not modify the set at all and will cause add() to return false. • Sets can be split and merged. • You can get the entire set of elements greater than or less than a target, for example. • Or you can merge two disjoint sets together. • This is called a union operation. • TreeSets are implemented using Binary Search Trees in Java, providing O(log n) insertion, access, and update and guaranteeing sorted order (remember to implement Comparable in your classes). • HashSets are implemented using hash tables, providing average-case O(1) insertion, access, and deletion, but not guaranteeing sorted order. • They are thus appropriate data structures to use for operations such as picking out the unique words in a book and outputting them in sorted order.

21. Methods • Has some of the usual ones: add(), remove(), size(), addAll()… • But also some exotic ones that return elements or subsets greater than or less than an element: • higher(Object): Returns the first element > Object. • lower(Object): Returns the first element < Object. • floor(Object): Returns the first element ≤ Object. • ceiling(Object): Returns the first element ≥ Object. • headSet(Object): Returns the whole set < Object. • tailSet(Object): Returns the whole set > Object. • Iterating over sets: • Use java’s “for each loop”: • Set<Type> dataset; • for (Type t : dataset) • System.out.println(t); //Or whatever you want to do with it. • Maps give you two sets: keySet() and values(). You can iterate over these too: Map<Key,Val> datamap; for(Key k : datamap.keySet())

22. Performance on a Sequence • We covered amortized analysis and sets today. • Next time, we will discuss graphs – the root data structure from which most others derive. • The lesson: • Plan for the future. Plan your current actions to make your future efforts easier.