Introduction to algorithm
Download
1 / 76

Introduction to Algorithm - PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on

Introduction to Algorithm. Algorithm: . Well-defined computational procedure that takes some value(s) (called input) and produces some value(s) (called output). Or a sequence of computational steps that transforms the input into the output . Example: sorting problem.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Introduction to Algorithm' - brooklyn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Introduction to algorithm
Introduction to Algorithm

Algorithm:

Well-defined computational procedure that

takes some value(s) (called input) and produces

some value(s) (called output).

Or a sequence of computational steps that

transforms the input into the output.

Example: sorting problem.

Input: a sequence of n number a1, a2, …,an

Output: a permutation (reordering) a1', a2', …,an'

such that a1' a2'  …  an '.

Problem instance: <31,41,59,26,41,58> <26,31,41,41,58,59>

Correctness: an algorithm is said to be correct if for every

input instance, it halts with correct output.


Algorithm design and analysis
Algorithm Design and Analysis

  • Data structure: the way to store and organize data.

  • Techniques:

    • different design and analysis techniques in this course.

    • purpose is for you to design/analyze your own algorithms for new problems in the future.

  • Hard problem:

    • Most problems discussed are efficient (poly time)

    • An interesting set of hard problems: NP-complete.


Np complete problem
NP-complete problem

  • Why interesting:

    • Not known whether an efficient algorithm exists for them.

    • If exist for one, then exist for all.

    • A small change may cause big change.

  • Why important:

    • Arise surprisingly often in real world.

    • Not waste time on trying to find efficient algorithm to get best solution, instead find approximate or near-optimal solution.

  • Example: traveling-salesman problem.


Efficiency of algorithms
Efficiency of Algorithms

  • Algorithms for solving the same problem can differ dramatically in their efficiency.

  • much more significant than the differences due to hardware and software.

  • Comparison of two sorting algorithms (n=106 numbers):

    • Insertion sort: c1n2

    • Merge sort: c2n (lg n)

    • Best programmer (c1=2), machine language, one billion/second computer.

    • Bad programmer (c2=50), high-language, ten million/second computer.

    • 2 (106)2 instructions/109 instructions per second = 2000 seconds.

    • 50 (106 lg 106) instructions/107 instructions per second  100 seconds.

    • Thus, merge sort on B is 20 times faster than insertion sort on A!

    • If sorting ten million number, 2.3 days VS. 20 minutes.


Algorithms as technology
Algorithms as technology

  • Other modern technologies:

    • Hardware with high clock rate, pipeline, ….

    • Easy-to-use, GUI

    • Object-oriented systems

    • LAN and WAN

  • Algorithms are no less important than these.

  • Moreover, all these new technologies involve heavily algorithms.

  • More large and complicate problems

  • As a result, one characteristic separating truly skilled programmer from novices.


Why the course is important
Why the course is important

  • Algorithms as a technology

    • Suppose computers were infinitely fast and memory was free, is there any need to study algorithm?

    • Need to find the easiest one to implement.

    • May be very fast, but not infinitely.

    • Memory may be very cheap, but not free.

    • The resources (cpu time and memory) need to be used wisely, (efficient) algorithms will help.

  • As the basis of computer science discipline.

  • Core course in computer science/engineering department

  • (Put emphasis on practical aspects of algorithms).


Insertion sort algorithm
Insertion Sort Algorithm

Input: a sequence of n number a1, a2, …,an

Output: a permutation (reordering) a1', a2', …,an'

such that a1' a2'  …  an '.

  • Idea: (similar to sort a hand of playing cards)

    • Every time, take one card, and insert the card to correct position in already sorted cards.



Insertion sort algorithm cont
Insertion Sort Algorithm (cont.)

INSERTION-SORT(A)

  • forj = 2 to length[A]

  • dokey A[j]

  • //insert A[j] to sorted sequence A[1..j-1]

  • i  j-1

  • whilei >0and A[i]>key

  • do A[i+1]  A[i] //move A[i] one position right

  • i i-1

  • A[i+1]  key


Example of insertion sort
Example of Insertion Sort


Correctness of insertion sort algorithm
Correctness of Insertion Sort Algorithm

  • Loop invariant

    • At the start of each iteration of the for loop, the subarray A[1..j-1] contains original A[1..j-1] but in sorted order.

  • Proof:

    • Initialization : j=2, A[1..j-1]=A[1..1]=A[1], sorted.

    • Maintenance: each iteration maintains loop invariant.

    • Termination: j=n+1, so A[1..j-1]=A[1..n] in sorted order.

  • Pseudocode conventions


Analyzing algorithms
Analyzing algorithms

  • Time (not memory, bandwidth, hardware)

  • Implementation model

    • Random-Access-Model (RAM)

      • One processor, sequential execution, no concurrency.

      • Basic data types

      • Basic operations (constant time per operation)

    • (Parallel multi-processor access model: PRAM)


Analysis of insertion sort
Analysis of Insertion Sort

INSERTION-SORT(A) costtimes

  • forj = 2 to length[A] c1n

  • dokey A[j] c2n-1

  • //insert A[j] to sorted sequence A[1..j-1] 0n-1

  • i  j-1 c4n-1

  • whilei >0and A[i]>keyc5j=2n tj

  • do A[i+1]  A[i] c6j=2n(tj –1)

  • i  i-1 c7j=2n(tj –1)

  • A[i+1]  key c8n–1

    (tj is the number of times the while loop test in line 5 is executed for that value of j)

    The total time cost T(n) = sum of cost times in each line

    =c1n + c2(n-1) + c4(n-1) + c5j=2n tj+ c6j=2n (tj-1)+ c7j=2n (tj-1)+ c8(n-1)


Analysis of insertion sort cont
Analysis of Insertion Sort (cont.)

  • Best case cost: already ordered numbers

    • tj=1, and line 6 and 7 will be executed 0 times

    • T(n) = c1n + c2(n-1) + c4(n-1) + c5(n-1) + c8(n-1)

      =(c1 + c2 + c4 + c5 + c8)n – (c2 + c4 + c5 + c8) = cn + c‘

  • Worst case cost: reverse ordered numbers

    • tj=j,

    • so j=2n tj = j=2n j =n(n+1)/2-1, and j=2n(tj –1) = j=2n(j–1) = n(n-1)/2, and

    • T(n) = c1n + c2(n-1) + c4(n-1) + c5(n(n+1)/2 -1) + + c6(n(n-1)/2 -1) + c7(n(n-1)/2)+ c8(n-1) =((c5 + c6 + c7)/2)n2 +(c1 + c2 + c4 +c5/2-c6/2-c7/2+c8)n-(c2 + c4 + c5 + c8) =an2+bn+c

  • Average case cost: random numbers

    • in average, tj = j/2. T(n) will still be in the order of n2, same as worst case.


The worst case running time
The Worst Case Running Time

  • An upper bound for any input.

  • For some algorithms, worst case occurs often.

  • Average case is often roughly as bad as the worst case.


Order of growth
Order of growth

  • Lower order item(s) are ignored, just keep the highest order item.

  • The constant coefficient(s) are ignored.

  • The rate of growth, or the order of growth, possesses the highest significance.

  • Use (n2) to represent the worst case running time for insertion sort.

  • (1), (lg n), (n),(n), (nlg n), (n2), (n3), (2n), (n!)


Design approaches
Design Approaches

  • Recursive algorithm

    • To solve a given problem, the algorithm recursively calls itself one or more time to deal with closely related subproblems.

    • f(n) = 0 if n=1

      1 if n=2

      f(n-1) + f(n-2), if n>2

  • Divide-and-conquer

  • Dynamic programming

  • Prune-and-search

  • Greedy algorithms, Linear programming, Parallel algorithms, Approximate algorithms, etc.


Divide and conquer approach
Divide-and-conquer approach

  • Divide the problem into a number of subproblems, with the same type as the original problem type.

  • Conquer the subproblems by solving them recursively. If the size of a subproblem is small enough, just solve the subproblem directly.

  • Combine the solutions to the subproblems into the solution for the original problem.


Merge sort
Merge Sort

  • Divide: divide the n-element sequence into two subproblems of n/2 elements each.

  • Conquer: sort the two subsequences recursively using merge sort. If the length of a sequence is 1, do nothing since it is already in order.

  • Combine: merge the two sorted subsequences to produce the sorted answer.


Merge sort merge function
Merge Sort –merge function

  • Merge is the key operation in merge sort.

  • Suppose the (sub)sequence(s) are stored in the array A. moreover, A[p..q] and A[q+1..r] are two sorted subsequences.

  • MERGE(A,p,q,r) will merge the two subsequences into sorted sequence A[p..r]


Merge a p q r
MERGE(A,p,q,r)

  • n1 q – p + 1

  • n2 r – q

  • //createtemporaryL[1..n1+1], R[1..n2+1]

  • fori  1 ton1

  • do L[i]  A[p+i-1]

  • forj  1 ton2

  • do R[j]  A[q+j]

  • L[n1 +1]  

  • R[n2 +1]  

  • i  1

  • j  1

  • Fork  ptor

  • do if L[i]  R[j]

  • then A[k]  L[i]

  • i  i+1

  • else A[k]  R[j]

  • j  j+1




Correctness of merge a p q r
Correctness of MERGE(A,p,q,r)

  • Loop invariant

    • At start of each iteration of for loop of line 12-17, subarray A[p..k-1] contains the k-p smallest elements of L[1..n1+1] and R[1..n2+1] in sorted order. L[i] and R[j] are smallest elements of their arrays that have not been copied back to A.

  • Proof:

    • Initialization: k-p=0, i=j=1, L[1] and R[1] are smallest.

    • Maintenance: loop will maintain the invariant

    • Termination: k=r+1, so A[p..k-1]= A[p..r] contains all and in sorted.


Analysis of merge a p q r
Analysis of MERGE(A,p,q,r)

  • n1 q – p + 1 (1)

  • n2 r – q (1)

  • //createtemporaryL[1..n1+1], R[1..n2+1] (0)

  • fori  1 ton1 (n1) (n1 +1 times)

  • do L[i]  A[p+i-1] (1)* (n1) (n1 times)

  • forj  1 ton2 (n2) (n2 +1 times)

  • do R[j]  A[q+j] (1) * (n2) (n2 times)

  • L[n1 +1]   (1)

  • R[n2 +1]   (1)

  • i  1 (1)

  • j  1 (1)

  • Fork  ptor (n), suppose n = r-p+1

  • do if L[i]  R[j] (1)*(n)

  • then A[k]  L[i] (1)*O(n)

  • i  i+1 (1)*O(n)

  • else A[k]  R[j] (1)*O(n)

  • j  j+1 (1)*O(n)

    Thus the total running time is (n1)+ (n2)+ (n)+7 (1) = (n).


Merge sort a p r
MERGE-SORT(A,p,r)

  • ifp < r

  • thenq (p+r)/2

  • MERGE-SORT(A,p,q)

  • MERGE-SORT(A,q+1,r)

  • MERGE(A,p,q,r)

    Call to MERGE-SORT(A,1,n) (suppose n=length(A))



Analysis of divide and conquer
Analysis of Divide-and-Conquer

  • Described by recursive equation

  • Suppose T(n) is the running time on a problem of size n.

  • T(n) = (1) if nc

    aT(n/b)+D(n)+C(n) if n>c

    Where a: number of subproblems

    n/b: size of each subproblem

    D(n): cost of divide operation

    C(n): cost of combination operation


Analysis of merge sort
Analysis of MERGE-SORT

  • Divide: D(n) = (1)

  • Conquer: a=2,b=2, so 2T(n/2)

  • Combine: C(n) = (n)

  • T(n) = (1) if n=1

    2T(n/2)+ (n) if n>1

  • T(n) = c if n=1

    2T(n/2)+ cn if n>1


Compute t n by recursive tree
Compute T(n) by Recursive Tree

  • The recursive equation can be solved by recursive tree.

  • T(n) = 2T(n/2)+ cn, (See its Recursive Tree).

  • lg n+1 levels, cn at each level, thus

  • Total cost for merge sort is

    • T(n) =cnlg n +cn = (nlg n).

  • In contrast, insertion sort is

    • T(n) = (n2).


Recursion tree of t n 2 t n 2 cn
Recursion tree of T(n)=2T(n/2)+cn


Growth of functions
Growth of functions

  • Asymptotic efficiency:

    • Input size is very large

    • Ignore lower-order terms and constant coefficients

    • Only the highest order item, (order of growth).

  • Typical order of growth:

    • (1), (lg n), (n),(n), (nlg n), (n2), (n3), (2n), (n!)

  • Asymptotic notations: O, , , o, .


Notation
-notation

  • For a given function g(n),

    • (g(n))={f(n): there exist positive constants c1,c2,and n0 such that 0 c1g(n)  f(n)  c2g(n) for all n n0}

    • Write f(n)  (g(n)), or simply f(n) = (g(n)).

c2g(n)

f(n)

c1g(n)

n

n0

f(n) = (g(n))


Some g n functions
Some (g(n)) functions

  • Prove 1/2n2 – 3n = (n2).

    • c1 n2  1/2n2 – 3n  c2 n2

    • c1 1/2 – 3/n  c2

    • c2  ½ for any n 1, and c1 1/14 for any n 7

    • Thus, c1= 1/14, c2 = ½ , n0=7.

  • Prove 6n3 (n2).

  • In general, f(n)=an2+bn+c = (n2), where a,b,c are constants and a>0.

  • p(n) = i=0daini, where ai are constants and ad>0.

    • then p(n) = (nd).

  • (n0) or (1), constant time complexity.


O notation
O-notation

  • For a given function g(n),

    • O(g(n))={f(n): there exist positive constants c and n0 such that 0 f(n)  cg(n) for all n n0}

    • Write f(n)  O(g(n)), or simply f(n) = O(g(n)).

cg(n)

f(n)

n

n0

f(n) = O(g(n))


Relation between g n and o g n
Relation between (g(n)) and O(g(n))

  • f(n) = (g(n)) implies f(n) = O(g(n))

  • -notation is a stronger notion than O-notation.

  • (g(n)) O(g(n))

  • an2+bn+c = (n2), also =O(n2)

  • A little surprising: an+b = O(n2). Verify it!!!

  • Even n =O(n2) !!!

  • O-notation, represents asymptotic upper bound.

  • -notation, represents asymptotic tight bound.

  • Note for distinctions:

    • When we say worst-case running time for insertion sort is O(n2), which applies to all inputs, even an already sorted input runs only in O(n) time because O(n) O(n2).

    • However, worst-case running time (n2) for insertion sort does not apply to every input, since for an already sorted input, (n) (n2) .


Notation1
-notation

  • For a given function g(n),

    • (g(n))={f(n): there exist positive constants c and n0 such that 0  cg(n)  f(n) for all n n0}

    • Write f(n)  (g(n)), or simply f(n) = (g(n)).

f(n)

cg(n)

n

n0

f(n) = (g(n))


Relation among o notations
Relation among O, ,-notations

  • Theorem 3.1 (page 46)

    • For f(n) and g(n), f(n) = (g(n)) if and only if f(n) = O(g(n)) and f(n) = (g(n)).

  • O-notation: asymptotic upper bound.

  • -notation: asymptotic tight bound.

  • -notation: asymptotic lower bound.


Notes on notation
Notes on -notation

  • generally used for best case running time.

  • So correct for all inputs.

  • For example, in insertion sort

    • best case running time is (n),

    • or correctly say: running time is (n).

    • It is OK to say worst case running time is (n2)

    • However, it is incorrect to say running time is (n2).


O notation1
o-notation

  • For a given function g(n),

    • o(g(n))={f(n): for any positive constant c,there exist a positive n0 such that 0 f(n)  cg(n) for all n n0}

    • Write f(n)  o(g(n)), or simply f(n) = o(g(n)).

2g(n)

g(n)

1/2g(n)

f(n)

n

n0

n0

n0

f(n) = o(g(n))


Notes on o notition

g(n)

n

Notes on o-notition

  • O-notation may or may not be asymptotically tight for upper bound.

    • 2n2 = O(n2) is tight, but 2n = O(n2) is not tight.

  • o-notition is used to denote an upper bound that is not asymptotically tight.

    • 2n = o(n2), but 2n2o(n2).

  • Difference: for some positive constant c in O-notation, but all positive constants c in o-notation.

  • In o-notation, f(n) becomes insignificant relative to g(n) as n approaches infinitely: i.e.,

    • lim = 0.

f(n)


Notation2
-notation

  • For a given function g(n),

    • (g(n))={f(n): for any positive constant c, there exists a positive n0 such that 0  cg(n)  f(n) for all n n0}

    • Write f(n)  (g(n)), or simply f(n) = (g(n)).

  • -notation, similar to o-notation, denotes lower bound that is not asymptotically tight.

    • n2/2 = (n), but n2/2  (n2)

  • f(n) = (g(n)) if and only if g(n)=o(f(n)).

  • lim = 

f(n)

g(n)

n


Properties of asymptotic notations
Properties of asymptotic notations

  • Transitivity: all five notations

    • f(n)= (g(n)) and g(n)= (h(n))  f(n)= (h(n))

  • Reflexivity: O, ,  notations.

    • f(n)= (f(n))

  • Symmetry:  notation.

    • f(n)= (g(n)) if and only if g(n)= (f(n))

  • Transpose symmetry:

    • f(n) = O(g(n)) if and only if g(n)= (f(n)).

    • f(n) = o(g(n)) if and only if g(n)= (f(n)).


Asymptotic efficiency of recurrences
Asymptotic Efficiency of Recurrences

  • Find the asymptotic bounds of recursive equations.

    • Substitution method

    • Recursive tree method

    • Master method (master theorem)

      • Provides bounds for: T(n) = aT(n/b)+f(n) where

        • a 1 (the number of subproblems).

        • b>1, (n/b is the size of each subproblem).

        • f(n) is a given function.


Recurrences
Recurrences

  • MERGE-SORT

    • T(n) = (1) if n=1

      2T(n/2)+ (n) if n>1

    • Ignore certain details:

      • T(n) = (1) if n=1

        T(n/2)+ T(n/2)+ (n) if n>1

      • Moreover, T(n) = 2T(n/2)+ (n)


The substitution method
The Substitution Method

  • Two steps:

    • Guess the form of the solution.

      • By experience, and creativity.

      • By Some heuristics.

        • If a recurrence is similar to one you have seen before.

          • T(n)=2T(n/2+17)+n, similar to T(n)=2T(n/2)+n, , guess O(nlg n).

        • Prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty.

          • For T(n)=2T(n/2)+n, prove lower bound T(n)= (n), and prove upper bound T(n)= O(n2), then guess the tight bound is T(n)=O(nlg n).

      • By recursion tree.

    • Use mathematical induction to find the constants and show that the solution works.


Solve t n 2 t n 2 n
Solve T(n)=2T(n/2)+n

  • Guess the solution: T(n)=O(nlg n),

    • i.e., T(n)cnlg n for some c.

  • Prove the solution by induction:

    • Suppose this bound holds for n/2, i.e.,

      • T(n/2)cn/2lg (n/2).

    • T(n)  2(cn/2lg (n/2))+n

      •  cnlg (n/2))+n

      • = cnlg n - cnlg 2 +n

      • = cnlg n - cn +n

      •  cnlg n (as long as c1)

  • Is the above proof complete? Why?


Boundary base condition
Boundary (base) Condition

  • In fact, T(n) =1 if n=1, i.e., T(1)=1.

  • However, cnlg n =c1lg 1 = 0, which is odd with T(1)=1.

  • Take advantage of asymptotic notation: it is required T(n)cnlg n hold for n n0 where n0 is a constant of our choosing.

  • Select n0 =2, thus, n=2 and n=3 as our induction bases. It turns out any c 2 suffices for base cases of n=2 and n=3 to hold.


Subtleties
Subtleties

  • Guess is correct, but induction proof not work.

  • Problem is that inductive assumption not strong enough.

  • Solution: revise the guess by subtracting a lower-order term.

  • Example: T(n)=T(n/2)+T(n/2)+1.

    • Guess T(n)=O(n), i.e., T(n) cn for some c.

    • However, T(n) c n/2+c n/2+1 =cn+1, which does not imply T(n) cn for any c.

    • Attempting T(n)=O(n2) will work, but overkill.

    • New guess T(n) cn – b will work as long as b  1.


Avoiding pitfall
Avoiding Pitfall

  • It is easy to guess T(n)=O(n) (i.e., T(n) cn) for T(n)=2T(n/2)+n.

  • And wrongly prove:

    • T(n)  2(cn/2)+n

      •  cn+n

      • =O(n). wrongly !!!!

  • Problem is that it does not prove the exactform of T(n) cn.


Changing variables
Changing Variables

  • Suppose T(n)=2T(n)+lg n.

  • Rename m=lg n. so T(2m)=2T(2m/2)+m.

  • Rename S(m)=T(2m), so S(m)=2S(m/2)+m.

  • Which is similar to T(n)=2T(n/2)+n.

  • Sothe solution is S(m)=O(m lg m).

  • Changing back to T(n) from S(m), the solution is T(n) =T(2m)=S(m)=O(m lg m)=O(lg n lg lg n).


The recursion tree method
The Recursion-tree Method

  • Idea:

    • Each node represents the cost of a single subproblem.

    • Sum up the costs with each level to get level cost.

    • Sum up all the level costs to get total cost.

  • Particularly suitable for divide-and-conquer recurrence.

  • Best used to generate a good guess, tolerating “sloppiness”.

  • If trying to compute cost as exact as possible, then used as direct proof.


Recursion tree for t n 3 t n 4 n 2

cn2

c(n/4)2

c(n/4)2

c(n/4)2

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

T(n/16)

(c)

Recursion Tree for T(n)=3T(n/4)+(n2)

T(n)

cn2

T(n/4)

T(n/4)

T(n/4)

(a)

(b)

cn2

cn2

(3/16)cn2

c(n/4)2

c(n/4)2

c(n/4)2

log 4n

(3/16)2cn2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

c(n/16)2

(nlog 43)

T(1)

T(1)

T(1)

T(1)

T(1)

T(1)

nlog 43

Total O(n2)

(d)


Solution to t n 3 t n 4 n 2
Solution to T(n)=3T(n/4)+(n2)

  • The height is log 4n,

  • #leaf nodes = 3log 4n= nlog 43. Leaf node cost: T(1).

  • Total cost T(n)=cn2+(3/16) cn2+(3/16)2cn2+

    …+(3/16)log 4n-1cn2+ (nlog 43)

    =(1+3/16+(3/16)2+…+(3/16)log 4n-1) cn2 + (nlog 43)

    <(1+3/16+(3/16)2+…+(3/16)m+…) cn2 + (nlog 43)

    =(1/(1-3/16)) cn2 + (nlog 43)

    =16/13cn2 + (nlog 43)

    =O(n2).


Prove the above guess
Prove the above Guess

  • T(n)=3T(n/4)+(n2) =O(n2).

  • Show T(n) dn2 for some d.

  • T(n) 3(d (n/4)2) +cn2

    3(d (n/4)2) +cn2

    =3/16(dn2) +cn2

     dn2, as long as d(16/13)c.


One more example
One more example

  • T(n)=T(n/3)+ T(2n/3)+O(n).

  • Construct its recursive tree (Figure 4.2, page 71).

  • T(n)=O(cnlg3/2n) = O(nlg n).

  • Prove T(n)  dnlg n.


Recursion tree of t n t n 3 t 2 n 3 o n
Recursion Tree of T(n)=T(n/3)+ T(2n/3)+O(n)


Master method theorem
Master Method/Theorem

  • Theorem 4.1 (page 73)

    • for T(n) = aT(n/b)+f(n), n/b may ben/born/b.

    • where a 1, b>1, f(n) be a function.

    • If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

    • If f(n)= (nlogba), then T(n)= (nlogba lg n).

    • If f(n)=(nlogba+) for some >0, and if af(n/b) cf(n) for some c<1 and all sufficiently large n, then T(n)= (f(n)).


Implications of master theorem
Implications of Master Theorem

  • Comparison between f(n) and nlogba (<,=,>)

  • Must be asymptotically smaller (or larger) by a polynomial, i.e., n for some >0.

  • In case 3, the “regularity” must be satisfied, i.e., af(n/b) cf(n) for some c<1 .

  • There are gaps

    • between 1 and 2: f(n) is smaller than nlogba, but not polynomially smaller.

    • between 2 and 3: f(n) is larger than nlogba, but not polynomially larger.

    • in case 3, if the “regularity” fails to hold.


Application of master theorem
Application of Master Theorem

  • T(n) = 9T(n/3)+n;

    • a=9,b=3, f(n) =n

    • nlogba = nlog39 =  (n2)

    • f(n)=O(nlog39-) for =1

    • By case 1, T(n) = (n2).

  • T(n) = T(2n/3)+1

    • a=1,b=3/2, f(n) =1

    • nlogba = nlog3/21 =  (n0) =  (1)

    • By case 2, T(n)= (lg n).


Application of master theorem1
Application of Master Theorem

  • T(n) = 3T(n/4)+nlg n;

    • a=3,b=4, f(n) =nlg n

    • nlogba = nlog43 =  (n0.793)

    • f(n)= (nlog43+) for 0.2

    • Moreover, for large n, the “regularity” holds for c=3/4.

      • af(n/b) =3(n/4)lg(n/4)  (3/4)nlg n =cf(n)

    • By case 3, T(n) = (f(n))= (nlg n).


Exception to master theorem
Exception to Master Theorem

  • T(n) = 2T(n/2)+nlg n;

    • a=2,b=2, f(n) =nlg n

    • nlogba = nlog22 =  (n)

    • f(n) is asymptotically larger than nlogba , but not polynomially larger because

    • f(n)/nlogba = lg n, which is asymptotically less than n for any >0.

    • Therefore,this is a gap between 2 and 3.


Where are the gaps
Where Are the Gaps

f(n), case 3, at least polynomially larger

n

Gap between case 3 and 2

c1

f(n), case 2: within constant distances

nlogba

c2

n

Gap between case 1 and 2

f(n), case 1, at least polynomially smaller

Note: 1. for case 3, the regularity also must hold.

2. if f(n) is lg n smaller, then fall in gap in 1 and 2

3. if f(n) is lg n larger, then fall in gap in 3 and 2

4. if f(n)=(nlogbalgkn), then T(n)=(nlogbalgk+1n). (as exercise)


Proof of master theorem

logbn-1

j=0

Proof of Master Theorem

  • The proof for the exact powers, n=bk for k1.

  • Lemma 4.2

    • for T(n) = (1) if n=1

    • aT(n/b)+f(n) if n=bk for k1

    • where a 1, b>1, f(n) be a nonnegative function,

    • Then

    • T(n) = (nlogba)+ ajf(n/bj)

  • Proof:

    • By iterating the recurrence

    • By recursion tree (See figure 4.3)


Recursion tree for t n at n b f n
Recursion tree for T(n)=aT(n/b)+f(n)


Proof of master theorem cont

logbn-1

j=0

Proof of Master Theorem (cont.)

  • Lemma 4.3:

    • Let a 1, b>1, f(n) be a nonnegative function defined on exact power of b, then

    • g(n)= ajf(n/bj) can be bounded for exact power of b as:

    • If f(n)=O(nlogba-) for some >0, then g(n)= O(nlogba).

    • If f(n)= (nlogba), then g(n)= (nlogba lg n).

    • If af(n/b) cf(n) for some c<1 and all sufficiently large n b, then g(n)= (f(n)).


Proof of lemma 4 3

logbn-1

logbn-1

logbn-1

logbn-1

logbn-1

j=0

j=0

j=0

j=0

j=0

Proof of Lemma 4.3

  • For case 1: f(n)=O(nlogba-) implies f(n/bj)=O((n /bj)logba-), so

  • g(n)=ajf(n/bj) =O(aj(n /bj)logba- )

  • = O(nlogba- aj/(blogba-)j ) = O(nlogba- aj/(aj(b-)j))

  • = O(nlogba- (b)j ) = O(nlogba- (((b )logbn-1)/(b-1) )

  • = O(nlogba- (((blogbn)-1)/(b-1)))=O(nlogba n-(n-1)/(b-1))

  • = O(nlogba )


Proof of lemma 4 3 cont

logbn-1

logbn-1

logbn-1

logbn-1

j=0

j=0

j=0

j=0

Proof of Lemma 4.3(cont.)

  • For case 2: f(n)= (nlogba) implies f(n/bj)= ((n /bj)logba), so

  • g(n)=ajf(n/bj) = (aj(n /bj)logba)

  • = (nlogbaaj/(blogba)j ) = (nlogba1)

  • = (nlogba logbn) = (nlogbalg n)


Proof of lemma 4 3 cont1

logbn-1

logbn-1

j=0

j=0

Proof of Lemma 4.3(cont.)

  • For case 3:

    • Since g(n) contains f(n), g(n) = (f(n))

    • Since af(n/b) cf(n), ajf(n/bj) cjf(n) , why???

    • g(n)= ajf(n/bj)  cjf(n)  f(n)cj

    • =f(n)(1/(1-c)) =O(f(n))

    • Thus, g(n)=(f(n))

j=0


Proof of master theorem cont1
Proof of Master Theorem (cont.)

  • Lemma 4.4:

    • for T(n) = (1) if n=1

    • aT(n/b)+f(n) if n=bk for k1

    • where a 1, b>1, f(n) be a nonnegative function,

    • If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

    • If f(n)= (nlogba), then T(n)= (nlogba lg n).

    • If f(n)=(nlogba+) for some >0, and if af(n/b) cf(n) for some c<1 and all sufficiently large n, then T(n)= (f(n)).


Proof of lemma 4 4 cont
Proof of Lemma 4.4 (cont.)

  • Combine Lemma 4.2 and 4.3,

    • For case 1:

      • T(n)= (nlogba)+O(nlogba)=(nlogba).

    • For case 2:

      • T(n)= (nlogba)+(nlogba lg n)=(nlogba lg n).

    • For case 3:

      • T(n)= (nlogba)+(f(n))=(f(n)) because f(n)= (nlogba+).


Floors and ceilings
Floors and Ceilings

  • T(n)=aT(n/b)+f(n) and T(n)=aT(n/b)+f(n)

  • Want to prove both equal T(n)=aT(n/b)+f(n)

  • Two results:

    • Master theorem applied to all integers n.

    • Floors and ceilings does not change the result.

  • Since n/bn/b, and n/b n/b, upper bound for floors and lower bound for ceiling is held.

  • So prove upper bound for ceilings (similar for lower bound for floors).


Upper bound of proof for t n at n b f n
Upper bound of proof for T(n)=aT(n/b)+f(n)

  • consider sequence n, n/b, n/b/b,  n/b/b/b, …

  • Let us define nj as follows:

  • nj = n if j=0

  • = nj-1/bif j>0

  • The sequence will be n0, n1, …, nlogbn

  • Draw recursion tree:


Recursion tree of t n at n b f n
Recursion tree of T(n)=aT(n/b)+f(n)


The proof of upper bound for ceiling

logbn-1

j=0

The proof of upper bound for ceiling

  • T(n) = (nlogba)+ ajf(nj)

  • Thus similar to Lemma 4.3 and 4.4, the upper bound is proven.


Summary
Summary

  • Analysis of running time and space

    • Worst case running time.

    • Average case (expected) running time.

      • Amortized analysis and amortized cost.

    • Best cast running time.

  • Loop invariant and its proof.

  • Asymptotic notations, definitions and relations.

  • Recurrences and their bounds

    • Substitution

    • Recursion tree

    • Master theorem.

    • Proof of subtleties and recurrence Master theorem does not apply to.


ad