1 / 76

Introduction to Algorithm - PowerPoint PPT Presentation

Introduction to Algorithm. Algorithm: . Well-defined computational procedure that takes some value(s) (called input) and produces some value(s) (called output). Or a sequence of computational steps that transforms the input into the output . Example: sorting problem.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

PowerPoint Slideshow about 'Introduction to Algorithm' - brooklyn

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Algorithm:

Well-defined computational procedure that

takes some value(s) (called input) and produces

some value(s) (called output).

Or a sequence of computational steps that

transforms the input into the output.

Example: sorting problem.

Input: a sequence of n number a1, a2, …,an

Output: a permutation (reordering) a1', a2', …,an'

such that a1' a2'  …  an '.

Problem instance: <31,41,59,26,41,58> <26,31,41,41,58,59>

Correctness: an algorithm is said to be correct if for every

input instance, it halts with correct output.

• Data structure: the way to store and organize data.

• Techniques:

• different design and analysis techniques in this course.

• purpose is for you to design/analyze your own algorithms for new problems in the future.

• Hard problem:

• Most problems discussed are efficient (poly time)

• An interesting set of hard problems: NP-complete.

• Why interesting:

• Not known whether an efficient algorithm exists for them.

• If exist for one, then exist for all.

• A small change may cause big change.

• Why important:

• Arise surprisingly often in real world.

• Not waste time on trying to find efficient algorithm to get best solution, instead find approximate or near-optimal solution.

• Example: traveling-salesman problem.

• Algorithms for solving the same problem can differ dramatically in their efficiency.

• much more significant than the differences due to hardware and software.

• Comparison of two sorting algorithms (n=106 numbers):

• Insertion sort: c1n2

• Merge sort: c2n (lg n)

• Best programmer (c1=2), machine language, one billion/second computer.

• Bad programmer (c2=50), high-language, ten million/second computer.

• 2 (106)2 instructions/109 instructions per second = 2000 seconds.

• 50 (106 lg 106) instructions/107 instructions per second  100 seconds.

• Thus, merge sort on B is 20 times faster than insertion sort on A!

• If sorting ten million number, 2.3 days VS. 20 minutes.

• Other modern technologies:

• Hardware with high clock rate, pipeline, ….

• Easy-to-use, GUI

• Object-oriented systems

• LAN and WAN

• Algorithms are no less important than these.

• Moreover, all these new technologies involve heavily algorithms.

• More large and complicate problems

• As a result, one characteristic separating truly skilled programmer from novices.

• Algorithms as a technology

• Suppose computers were infinitely fast and memory was free, is there any need to study algorithm?

• Need to find the easiest one to implement.

• May be very fast, but not infinitely.

• Memory may be very cheap, but not free.

• The resources (cpu time and memory) need to be used wisely, (efficient) algorithms will help.

• As the basis of computer science discipline.

• Core course in computer science/engineering department

• (Put emphasis on practical aspects of algorithms).

Input: a sequence of n number a1, a2, …,an

Output: a permutation (reordering) a1', a2', …,an'

such that a1' a2'  …  an '.

• Idea: (similar to sort a hand of playing cards)

• Every time, take one card, and insert the card to correct position in already sorted cards.

INSERTION-SORT(A)

• forj = 2 to length[A]

• dokey A[j]

• //insert A[j] to sorted sequence A[1..j-1]

• i  j-1

• whilei >0and A[i]>key

• do A[i+1]  A[i] //move A[i] one position right

• i i-1

• A[i+1]  key

Example of Insertion Sort

• Loop invariant

• At the start of each iteration of the for loop, the subarray A[1..j-1] contains original A[1..j-1] but in sorted order.

• Proof:

• Initialization : j=2, A[1..j-1]=A[1..1]=A[1], sorted.

• Maintenance: each iteration maintains loop invariant.

• Termination: j=n+1, so A[1..j-1]=A[1..n] in sorted order.

• Pseudocode conventions

• Time (not memory, bandwidth, hardware)

• Implementation model

• Random-Access-Model (RAM)

• One processor, sequential execution, no concurrency.

• Basic data types

• Basic operations (constant time per operation)

• (Parallel multi-processor access model: PRAM)

INSERTION-SORT(A) costtimes

• forj = 2 to length[A] c1n

• dokey A[j] c2n-1

• //insert A[j] to sorted sequence A[1..j-1] 0n-1

• i  j-1 c4n-1

• whilei >0and A[i]>keyc5j=2n tj

• do A[i+1]  A[i] c6j=2n(tj –1)

• i  i-1 c7j=2n(tj –1)

• A[i+1]  key c8n–1

(tj is the number of times the while loop test in line 5 is executed for that value of j)

The total time cost T(n) = sum of cost times in each line

=c1n + c2(n-1) + c4(n-1) + c5j=2n tj+ c6j=2n (tj-1)+ c7j=2n (tj-1)+ c8(n-1)

• Best case cost: already ordered numbers

• tj=1, and line 6 and 7 will be executed 0 times

• T(n) = c1n + c2(n-1) + c4(n-1) + c5(n-1) + c8(n-1)

=(c1 + c2 + c4 + c5 + c8)n – (c2 + c4 + c5 + c8) = cn + c‘

• Worst case cost: reverse ordered numbers

• tj=j,

• so j=2n tj = j=2n j =n(n+1)/2-1, and j=2n(tj –1) = j=2n(j–1) = n(n-1)/2, and

• T(n) = c1n + c2(n-1) + c4(n-1) + c5(n(n+1)/2 -1) + + c6(n(n-1)/2 -1) + c7(n(n-1)/2)+ c8(n-1) =((c5 + c6 + c7)/2)n2 +(c1 + c2 + c4 +c5/2-c6/2-c7/2+c8)n-(c2 + c4 + c5 + c8) =an2+bn+c

• Average case cost: random numbers

• in average, tj = j/2. T(n) will still be in the order of n2, same as worst case.

• An upper bound for any input.

• For some algorithms, worst case occurs often.

• Average case is often roughly as bad as the worst case.

• Lower order item(s) are ignored, just keep the highest order item.

• The constant coefficient(s) are ignored.

• The rate of growth, or the order of growth, possesses the highest significance.

• Use (n2) to represent the worst case running time for insertion sort.

• (1), (lg n), (n),(n), (nlg n), (n2), (n3), (2n), (n!)

• Recursive algorithm

• To solve a given problem, the algorithm recursively calls itself one or more time to deal with closely related subproblems.

• f(n) = 0 if n=1

1 if n=2

f(n-1) + f(n-2), if n>2

• Divide-and-conquer

• Dynamic programming

• Prune-and-search

• Greedy algorithms, Linear programming, Parallel algorithms, Approximate algorithms, etc.

• Divide the problem into a number of subproblems, with the same type as the original problem type.

• Conquer the subproblems by solving them recursively. If the size of a subproblem is small enough, just solve the subproblem directly.

• Combine the solutions to the subproblems into the solution for the original problem.

• Divide: divide the n-element sequence into two subproblems of n/2 elements each.

• Conquer: sort the two subsequences recursively using merge sort. If the length of a sequence is 1, do nothing since it is already in order.

• Combine: merge the two sorted subsequences to produce the sorted answer.

• Merge is the key operation in merge sort.

• Suppose the (sub)sequence(s) are stored in the array A. moreover, A[p..q] and A[q+1..r] are two sorted subsequences.

• MERGE(A,p,q,r) will merge the two subsequences into sorted sequence A[p..r]

MERGE(A,p,q,r)

• n1 q – p + 1

• n2 r – q

• //createtemporaryL[1..n1+1], R[1..n2+1]

• fori  1 ton1

• do L[i]  A[p+i-1]

• forj  1 ton2

• do R[j]  A[q+j]

• L[n1 +1]  

• R[n2 +1]  

• i  1

• j  1

• Fork  ptor

• do if L[i]  R[j]

• then A[k]  L[i]

• i  i+1

• else A[k]  R[j]

• j  j+1

Correctness of MERGE(A,p,q,r)

• Loop invariant

• At start of each iteration of for loop of line 12-17, subarray A[p..k-1] contains the k-p smallest elements of L[1..n1+1] and R[1..n2+1] in sorted order. L[i] and R[j] are smallest elements of their arrays that have not been copied back to A.

• Proof:

• Initialization: k-p=0, i=j=1, L[1] and R[1] are smallest.

• Maintenance: loop will maintain the invariant

• Termination: k=r+1, so A[p..k-1]= A[p..r] contains all and in sorted.

Analysis of MERGE(A,p,q,r)

• n1 q – p + 1 (1)

• n2 r – q (1)

• //createtemporaryL[1..n1+1], R[1..n2+1] (0)

• fori  1 ton1 (n1) (n1 +1 times)

• do L[i]  A[p+i-1] (1)* (n1) (n1 times)

• forj  1 ton2 (n2) (n2 +1 times)

• do R[j]  A[q+j] (1) * (n2) (n2 times)

• L[n1 +1]   (1)

• R[n2 +1]   (1)

• i  1 (1)

• j  1 (1)

• Fork  ptor (n), suppose n = r-p+1

• do if L[i]  R[j] (1)*(n)

• then A[k]  L[i] (1)*O(n)

• i  i+1 (1)*O(n)

• else A[k]  R[j] (1)*O(n)

• j  j+1 (1)*O(n)

Thus the total running time is (n1)+ (n2)+ (n)+7 (1) = (n).

MERGE-SORT(A,p,r)

• ifp < r

• thenq (p+r)/2

• MERGE-SORT(A,p,q)

• MERGE-SORT(A,q+1,r)

• MERGE(A,p,q,r)

Call to MERGE-SORT(A,1,n) (suppose n=length(A))

• Described by recursive equation

• Suppose T(n) is the running time on a problem of size n.

• T(n) = (1) if nc

aT(n/b)+D(n)+C(n) if n>c

Where a: number of subproblems

n/b: size of each subproblem

D(n): cost of divide operation

C(n): cost of combination operation

• Divide: D(n) = (1)

• Conquer: a=2,b=2, so 2T(n/2)

• Combine: C(n) = (n)

• T(n) = (1) if n=1

2T(n/2)+ (n) if n>1

• T(n) = c if n=1

2T(n/2)+ cn if n>1

Compute T(n) by Recursive Tree

• The recursive equation can be solved by recursive tree.

• T(n) = 2T(n/2)+ cn, (See its Recursive Tree).

• lg n+1 levels, cn at each level, thus

• Total cost for merge sort is

• T(n) =cnlg n +cn = (nlg n).

• In contrast, insertion sort is

• T(n) = (n2).

Recursion tree of T(n)=2T(n/2)+cn

• Asymptotic efficiency:

• Input size is very large

• Ignore lower-order terms and constant coefficients

• Only the highest order item, (order of growth).

• Typical order of growth:

• (1), (lg n), (n),(n), (nlg n), (n2), (n3), (2n), (n!)

• Asymptotic notations: O, , , o, .

• For a given function g(n),

• (g(n))={f(n): there exist positive constants c1,c2,and n0 such that 0 c1g(n)  f(n)  c2g(n) for all n n0}

• Write f(n)  (g(n)), or simply f(n) = (g(n)).

c2g(n)

f(n)

c1g(n)

n

n0

f(n) = (g(n))

Some (g(n)) functions

• Prove 1/2n2 – 3n = (n2).

• c1 n2  1/2n2 – 3n  c2 n2

• c1 1/2 – 3/n  c2

• c2  ½ for any n 1, and c1 1/14 for any n 7

• Thus, c1= 1/14, c2 = ½ , n0=7.

• Prove 6n3 (n2).

• In general, f(n)=an2+bn+c = (n2), where a,b,c are constants and a>0.

• p(n) = i=0daini, where ai are constants and ad>0.

• then p(n) = (nd).

• (n0) or (1), constant time complexity.

O-notation

• For a given function g(n),

• O(g(n))={f(n): there exist positive constants c and n0 such that 0 f(n)  cg(n) for all n n0}

• Write f(n)  O(g(n)), or simply f(n) = O(g(n)).

cg(n)

f(n)

n

n0

f(n) = O(g(n))

Relation between (g(n)) and O(g(n))

• f(n) = (g(n)) implies f(n) = O(g(n))

• -notation is a stronger notion than O-notation.

• (g(n)) O(g(n))

• an2+bn+c = (n2), also =O(n2)

• A little surprising: an+b = O(n2). Verify it!!!

• Even n =O(n2) !!!

• O-notation, represents asymptotic upper bound.

• -notation, represents asymptotic tight bound.

• Note for distinctions:

• When we say worst-case running time for insertion sort is O(n2), which applies to all inputs, even an already sorted input runs only in O(n) time because O(n) O(n2).

• However, worst-case running time (n2) for insertion sort does not apply to every input, since for an already sorted input, (n) (n2) .

• For a given function g(n),

• (g(n))={f(n): there exist positive constants c and n0 such that 0  cg(n)  f(n) for all n n0}

• Write f(n)  (g(n)), or simply f(n) = (g(n)).

f(n)

cg(n)

n

n0

f(n) = (g(n))

Relation among O, ,-notations

• Theorem 3.1 (page 46)

• For f(n) and g(n), f(n) = (g(n)) if and only if f(n) = O(g(n)) and f(n) = (g(n)).

• O-notation: asymptotic upper bound.

• -notation: asymptotic tight bound.

• -notation: asymptotic lower bound.

Notes on -notation

• generally used for best case running time.

• So correct for all inputs.

• For example, in insertion sort

• best case running time is (n),

• or correctly say: running time is (n).

• It is OK to say worst case running time is (n2)

• However, it is incorrect to say running time is (n2).

o-notation

• For a given function g(n),

• o(g(n))={f(n): for any positive constant c,there exist a positive n0 such that 0 f(n)  cg(n) for all n n0}

• Write f(n)  o(g(n)), or simply f(n) = o(g(n)).

2g(n)

g(n)

1/2g(n)

f(n)

n

n0

n0

n0

f(n) = o(g(n))

g(n)

n

Notes on o-notition

• O-notation may or may not be asymptotically tight for upper bound.

• 2n2 = O(n2) is tight, but 2n = O(n2) is not tight.

• o-notition is used to denote an upper bound that is not asymptotically tight.

• 2n = o(n2), but 2n2o(n2).

• Difference: for some positive constant c in O-notation, but all positive constants c in o-notation.

• In o-notation, f(n) becomes insignificant relative to g(n) as n approaches infinitely: i.e.,

• lim = 0.

f(n)

• For a given function g(n),

• (g(n))={f(n): for any positive constant c, there exists a positive n0 such that 0  cg(n)  f(n) for all n n0}

• Write f(n)  (g(n)), or simply f(n) = (g(n)).

• -notation, similar to o-notation, denotes lower bound that is not asymptotically tight.

• n2/2 = (n), but n2/2  (n2)

• f(n) = (g(n)) if and only if g(n)=o(f(n)).

• lim = 

f(n)

g(n)

n

• Transitivity: all five notations

• f(n)= (g(n)) and g(n)= (h(n))  f(n)= (h(n))

• Reflexivity: O, ,  notations.

• f(n)= (f(n))

• Symmetry:  notation.

• f(n)= (g(n)) if and only if g(n)= (f(n))

• Transpose symmetry:

• f(n) = O(g(n)) if and only if g(n)= (f(n)).

• f(n) = o(g(n)) if and only if g(n)= (f(n)).

• Find the asymptotic bounds of recursive equations.

• Substitution method

• Recursive tree method

• Master method (master theorem)

• Provides bounds for: T(n) = aT(n/b)+f(n) where

• a 1 (the number of subproblems).

• b>1, (n/b is the size of each subproblem).

• f(n) is a given function.

• MERGE-SORT

• T(n) = (1) if n=1

2T(n/2)+ (n) if n>1

• Ignore certain details:

• T(n) = (1) if n=1

T(n/2)+ T(n/2)+ (n) if n>1

• Moreover, T(n) = 2T(n/2)+ (n)

• Two steps:

• Guess the form of the solution.

• By experience, and creativity.

• By Some heuristics.

• If a recurrence is similar to one you have seen before.

• T(n)=2T(n/2+17)+n, similar to T(n)=2T(n/2)+n, , guess O(nlg n).

• Prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty.

• For T(n)=2T(n/2)+n, prove lower bound T(n)= (n), and prove upper bound T(n)= O(n2), then guess the tight bound is T(n)=O(nlg n).

• By recursion tree.

• Use mathematical induction to find the constants and show that the solution works.

Solve T(n)=2T(n/2)+n

• Guess the solution: T(n)=O(nlg n),

• i.e., T(n)cnlg n for some c.

• Prove the solution by induction:

• Suppose this bound holds for n/2, i.e.,

• T(n/2)cn/2lg (n/2).

• T(n)  2(cn/2lg (n/2))+n

•  cnlg (n/2))+n

• = cnlg n - cnlg 2 +n

• = cnlg n - cn +n

•  cnlg n (as long as c1)

• Is the above proof complete? Why?

• In fact, T(n) =1 if n=1, i.e., T(1)=1.

• However, cnlg n =c1lg 1 = 0, which is odd with T(1)=1.

• Take advantage of asymptotic notation: it is required T(n)cnlg n hold for n n0 where n0 is a constant of our choosing.

• Select n0 =2, thus, n=2 and n=3 as our induction bases. It turns out any c 2 suffices for base cases of n=2 and n=3 to hold.

• Guess is correct, but induction proof not work.

• Problem is that inductive assumption not strong enough.

• Solution: revise the guess by subtracting a lower-order term.

• Example: T(n)=T(n/2)+T(n/2)+1.

• Guess T(n)=O(n), i.e., T(n) cn for some c.

• However, T(n) c n/2+c n/2+1 =cn+1, which does not imply T(n) cn for any c.

• Attempting T(n)=O(n2) will work, but overkill.

• New guess T(n) cn – b will work as long as b  1.

• It is easy to guess T(n)=O(n) (i.e., T(n) cn) for T(n)=2T(n/2)+n.

• And wrongly prove:

• T(n)  2(cn/2)+n

•  cn+n

• =O(n). wrongly !!!!

• Problem is that it does not prove the exactform of T(n) cn.

• Suppose T(n)=2T(n)+lg n.

• Rename m=lg n. so T(2m)=2T(2m/2)+m.

• Rename S(m)=T(2m), so S(m)=2S(m/2)+m.

• Which is similar to T(n)=2T(n/2)+n.

• Sothe solution is S(m)=O(m lg m).

• Changing back to T(n) from S(m), the solution is T(n) =T(2m)=S(m)=O(m lg m)=O(lg n lg lg n).

• Idea:

• Each node represents the cost of a single subproblem.

• Sum up the costs with each level to get level cost.

• Sum up all the level costs to get total cost.

• Particularly suitable for divide-and-conquer recurrence.

• Best used to generate a good guess, tolerating “sloppiness”.

• If trying to compute cost as exact as possible, then used as direct proof.

cn2

c(n/4)2

c(n/4)2

c(n/4)2

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

(c)

Recursion Tree for T(n)=3T(n/4)+(n2)

T(n)

cn2

T(n/4)

T(n/4)

T(n/4)

(a)

(b)

cn2

cn2

(3/16)cn2

c(n/4)2

c(n/4)2

c(n/4)2

log 4n

(3/16)2cn2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

(nlog 43)

T(1)

T(1)

T(1)

T(1)

T(1)

T(1)

nlog 43

Total O(n2)

(d)

Solution to T(n)=3T(n/4)+(n2)

• The height is log 4n,

• #leaf nodes = 3log 4n= nlog 43. Leaf node cost: T(1).

• Total cost T(n)=cn2+(3/16) cn2+(3/16)2cn2+

…+(3/16)log 4n-1cn2+ (nlog 43)

=(1+3/16+(3/16)2+…+(3/16)log 4n-1) cn2 + (nlog 43)

<(1+3/16+(3/16)2+…+(3/16)m+…) cn2 + (nlog 43)

=(1/(1-3/16)) cn2 + (nlog 43)

=16/13cn2 + (nlog 43)

=O(n2).

• T(n)=3T(n/4)+(n2) =O(n2).

• Show T(n) dn2 for some d.

• T(n) 3(d (n/4)2) +cn2

3(d (n/4)2) +cn2

=3/16(dn2) +cn2

 dn2, as long as d(16/13)c.

• T(n)=T(n/3)+ T(2n/3)+O(n).

• Construct its recursive tree (Figure 4.2, page 71).

• T(n)=O(cnlg3/2n) = O(nlg n).

• Prove T(n)  dnlg n.

Recursion Tree of T(n)=T(n/3)+ T(2n/3)+O(n)

• Theorem 4.1 (page 73)

• for T(n) = aT(n/b)+f(n), n/b may ben/born/b.

• where a 1, b>1, f(n) be a function.

• If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

• If f(n)= (nlogba), then T(n)= (nlogba lg n).

• If f(n)=(nlogba+) for some >0, and if af(n/b) cf(n) for some c<1 and all sufficiently large n, then T(n)= (f(n)).

• Comparison between f(n) and nlogba (<,=,>)

• Must be asymptotically smaller (or larger) by a polynomial, i.e., n for some >0.

• In case 3, the “regularity” must be satisfied, i.e., af(n/b) cf(n) for some c<1 .

• There are gaps

• between 1 and 2: f(n) is smaller than nlogba, but not polynomially smaller.

• between 2 and 3: f(n) is larger than nlogba, but not polynomially larger.

• in case 3, if the “regularity” fails to hold.

• T(n) = 9T(n/3)+n;

• a=9,b=3, f(n) =n

• nlogba = nlog39 =  (n2)

• f(n)=O(nlog39-) for =1

• By case 1, T(n) = (n2).

• T(n) = T(2n/3)+1

• a=1,b=3/2, f(n) =1

• nlogba = nlog3/21 =  (n0) =  (1)

• By case 2, T(n)= (lg n).

• T(n) = 3T(n/4)+nlg n;

• a=3,b=4, f(n) =nlg n

• nlogba = nlog43 =  (n0.793)

• f(n)= (nlog43+) for 0.2

• Moreover, for large n, the “regularity” holds for c=3/4.

• af(n/b) =3(n/4)lg(n/4)  (3/4)nlg n =cf(n)

• By case 3, T(n) = (f(n))= (nlg n).

• T(n) = 2T(n/2)+nlg n;

• a=2,b=2, f(n) =nlg n

• nlogba = nlog22 =  (n)

• f(n) is asymptotically larger than nlogba , but not polynomially larger because

• f(n)/nlogba = lg n, which is asymptotically less than n for any >0.

• Therefore,this is a gap between 2 and 3.

f(n), case 3, at least polynomially larger

n

Gap between case 3 and 2

c1

f(n), case 2: within constant distances

nlogba

c2

n

Gap between case 1 and 2

f(n), case 1, at least polynomially smaller

Note: 1. for case 3, the regularity also must hold.

2. if f(n) is lg n smaller, then fall in gap in 1 and 2

3. if f(n) is lg n larger, then fall in gap in 3 and 2

4. if f(n)=(nlogbalgkn), then T(n)=(nlogbalgk+1n). (as exercise)

logbn-1

j=0

Proof of Master Theorem

• The proof for the exact powers, n=bk for k1.

• Lemma 4.2

• for T(n) = (1) if n=1

• aT(n/b)+f(n) if n=bk for k1

• where a 1, b>1, f(n) be a nonnegative function,

• Then

• T(n) = (nlogba)+ ajf(n/bj)

• Proof:

• By iterating the recurrence

• By recursion tree (See figure 4.3)

Recursion tree for T(n)=aT(n/b)+f(n)

logbn-1

j=0

Proof of Master Theorem (cont.)

• Lemma 4.3:

• Let a 1, b>1, f(n) be a nonnegative function defined on exact power of b, then

• g(n)= ajf(n/bj) can be bounded for exact power of b as:

• If f(n)=O(nlogba-) for some >0, then g(n)= O(nlogba).

• If f(n)= (nlogba), then g(n)= (nlogba lg n).

• If af(n/b) cf(n) for some c<1 and all sufficiently large n b, then g(n)= (f(n)).

logbn-1

logbn-1

logbn-1

logbn-1

logbn-1

j=0

j=0

j=0

j=0

j=0

Proof of Lemma 4.3

• For case 1: f(n)=O(nlogba-) implies f(n/bj)=O((n /bj)logba-), so

• g(n)=ajf(n/bj) =O(aj(n /bj)logba- )

• = O(nlogba- aj/(blogba-)j ) = O(nlogba- aj/(aj(b-)j))

• = O(nlogba- (b)j ) = O(nlogba- (((b )logbn-1)/(b-1) )

• = O(nlogba- (((blogbn)-1)/(b-1)))=O(nlogba n-(n-1)/(b-1))

• = O(nlogba )

logbn-1

logbn-1

logbn-1

logbn-1

j=0

j=0

j=0

j=0

Proof of Lemma 4.3(cont.)

• For case 2: f(n)= (nlogba) implies f(n/bj)= ((n /bj)logba), so

• g(n)=ajf(n/bj) = (aj(n /bj)logba)

• = (nlogbaaj/(blogba)j ) = (nlogba1)

• = (nlogba logbn) = (nlogbalg n)

logbn-1

logbn-1

j=0

j=0

Proof of Lemma 4.3(cont.)

• For case 3:

• Since g(n) contains f(n), g(n) = (f(n))

• Since af(n/b) cf(n), ajf(n/bj) cjf(n) , why???

• g(n)= ajf(n/bj)  cjf(n)  f(n)cj

• =f(n)(1/(1-c)) =O(f(n))

• Thus, g(n)=(f(n))

j=0

• Lemma 4.4:

• for T(n) = (1) if n=1

• aT(n/b)+f(n) if n=bk for k1

• where a 1, b>1, f(n) be a nonnegative function,

• If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

• If f(n)= (nlogba), then T(n)= (nlogba lg n).

• If f(n)=(nlogba+) for some >0, and if af(n/b) cf(n) for some c<1 and all sufficiently large n, then T(n)= (f(n)).

• Combine Lemma 4.2 and 4.3,

• For case 1:

• T(n)= (nlogba)+O(nlogba)=(nlogba).

• For case 2:

• T(n)= (nlogba)+(nlogba lg n)=(nlogba lg n).

• For case 3:

• T(n)= (nlogba)+(f(n))=(f(n)) because f(n)= (nlogba+).

• T(n)=aT(n/b)+f(n) and T(n)=aT(n/b)+f(n)

• Want to prove both equal T(n)=aT(n/b)+f(n)

• Two results:

• Master theorem applied to all integers n.

• Floors and ceilings does not change the result.

• Since n/bn/b, and n/b n/b, upper bound for floors and lower bound for ceiling is held.

• So prove upper bound for ceilings (similar for lower bound for floors).

Upper bound of proof for T(n)=aT(n/b)+f(n)

• consider sequence n, n/b, n/b/b,  n/b/b/b, …

• Let us define nj as follows:

• nj = n if j=0

• = nj-1/bif j>0

• The sequence will be n0, n1, …, nlogbn

• Draw recursion tree:

Recursion tree of T(n)=aT(n/b)+f(n)

logbn-1

j=0

The proof of upper bound for ceiling

• T(n) = (nlogba)+ ajf(nj)

• Thus similar to Lemma 4.3 and 4.4, the upper bound is proven.

• Analysis of running time and space

• Worst case running time.

• Average case (expected) running time.

• Amortized analysis and amortized cost.

• Best cast running time.

• Loop invariant and its proof.

• Asymptotic notations, definitions and relations.

• Recurrences and their bounds

• Substitution

• Recursion tree

• Master theorem.

• Proof of subtleties and recurrence Master theorem does not apply to.