Loading in 2 Seconds...

COMP108 Algorithmic Foundations Algorithm efficiency + Searching/Sorting

Loading in 2 Seconds...

- 281 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'comp108 algorithmic foundations algorithm efficiency searching' - Gabriel

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### COMP108Algorithmic FoundationsAlgorithm efficiency + Searching/Sorting

### Some exponential time algorithms – Traveling Salesman Problem, Knapsack Problem …

Prudence Wong

http://www.csc.liv.ac.uk/~pwong/teaching/comp108

Learning outcomes

- Able to carry out simple asymptotic analysis of algorithms
- See some examples of polynomial time and exponential time algorithms
- Able to apply searching/sorting algorithms and derive their time complexities

Time Complexity Analysis

- How fast is the algorithm?

Code the algorithm and run the program, then measure the running time

- Depend on the speed of the computer
- Waste time coding and testing if the algorithm is slow

Identify some important operations/steps and count how many times these operations/steps needed to executed

Time Complexity Analysis

- How to measure efficiency?

Number of operations usually expressed in terms of input size

- If we doubled the input size, how much longer would the algorithm take?If we trebled the input size, how much longer would it take?

Why efficiency matters?

- speed of computation by hardware has been improved
- efficiency still matters
- ambition for computer applications grow with computer power
- demand a great increase in speed of computation

Amount of data handled matches speed increase?

- When computation speed vastly increased, can we handle much more data?

- Suppose
- an algorithm takes n2comparisons to sortn numbers
- we need 1 sec to sort 5 numbers (25 comparisons)
- computing speed increases by factor of100
- Using 1 sec, we can now perform100x25comparisons,i.e., to sort50 numbers
- With 100 times speedup, only sort 10 times more numbers!

Time Complexity Analysis

input n

sum = 0

for i = 1 to n do

begin

sum = sum + i

end

output sum

- Important operation of summation: addition
- How many additions this algorithm requires?

We need n additions

(depend on the input size n)

Space Complexity Analysis

input n

sum = 0

for i = 1 to n do

begin

sum = sum + i

end

output sum

We need to store the number n, the variable sum, and the index i

=> needs 3 memory space

In this example, the memory space does not depend on the input size n

In other case, it may depend on the input size n

Look for improvement

input n

sum = n*(n+1)/2

output sum

Mathematical formula gives us an alternative to find the sum of first n numbers:

1 + 2 + ... + n = n(n+1)/2

We only need 3 operations:

1 addition, 1 multiplication, and 1 division

(no matter what the input size n is)

Finding min among 3 numbers

Input: 3 numbers a, b, c

Output: the min value of these 3 numbers

Algorithm:

Input a, b, c

a b ?

N

Y

a c ?

Y

b c ?

N

Y

N

Output a

Output b

Output c

Output c

Finding min among 3 numbers

Example 1

Input a, b, c

a=3, b=5, c=2

Y

N

a b ?

a c ?

Y

b c ?

N

Y

N

Output a

Output b

Output c

Output c

2 is output

Finding min among 3 numbers

Example 2

Input a, b, c

a=6, b=5, c=7

Y

N

a b ?

a c ?

Y

b c ?

N

Y

N

Output a

Output b

Output c

Output c

5 is output

Time Complexity Analysis

- Important operation: comparison
- How many comparison this algorithm requires?

input a, b, c

if (a b) then

if (a c) then

output a

else

output c

else

if (b c) then

output b

else

output c

It takes 2 comparisons

Finding min among n numbers

Input a0, a1, ...

Any systematic approach?

a0 a1 ?

N

Y

a0 a2 ?

Y

a1 a2 ?

Y

N

N

a0 a3 ?

a1 a3 ?

a2 a3 ?

a2 a3 ?

Finding min among n numbers

Input a[0], a[1], ..., a[n-1]

min = 0

i = 1

i = i+1

i < n?

N

Y

Output a[min]

a[i] < a[min]?

Y

min = i

N

Finding min among n numbers

input a[0], a[1], ..., a[n-1]

min = 0

i = 1

while (i <n) do

begin

if (a[i] < a[min]) then

min = i

i = i + 1

end

output a[min]

Example

a[]={50,30,40,20,10}

0

50

1

30

1

30

3

20

4

10

Finding min among n numbers

n-1

- Time complexity: ?? comparisons

input a[0], a[1], ..., a[n-1]

min = 0

i = 1

while (i <n) do

begin

if (a[i] < a[min]) then

min = i

i = i + 1

end

output a[min]

Finding min using for-loop

- Rewrite the above while-loop into a for-loop

input a[0], a[1], ..., a[n-1]

min = 0

for i = 1 to n-1 do

begin

if (a[i] < a[min]) then

min = i

end

output a[min]

Which algorithm is the fastest?

- Consider a problem that can be solved by 5 algorithms A1, A2, A3, A4, A5 using different number of operations (time complexity).

f1(n) = 50n + 20

f2(n) = 10 n log2 n + 100

f3(n) = n2 - 3n + 6

f4(n) = 2n2

f5(n) = 2n/8 - n/4 + 2

f5(n)

f3(n)

f1(n)

Depends on the size of the problem!

What do we observe?

- There is huge difference between
- functions involving powers ofn (e.g., n, n log n, n2, called polynomial functions) and
- functions involving powering by n (e.g., 2n, called exponential functions)
- Among polynomial functions, those with same order of power are more comparable
- e.g., f3(n) = n2 - 3n + 6 and f4(n) = 2n2

Hierarchy of functions

- We can define a hierarchy of functions each having a greater order of magnitude than its predecessor:

1

log n

n n2 n3 ... nk ...

2n

constant

logarithmic

polynomial

exponential

- We can further refine the hierarchy by inserting n log n between n and n2,n2 log n between n2 and n3, and so on.

Hierarchy of functions (2)

1

log n

n n2 n3 ... nk ...

2n

- Important: as we move from left to right, successive functions have greater order of magnitude than the previous ones.
- As n increases, the values of the later functions increase more rapidly than the values of the earlier ones.
- Relative growth rates increase

constant

logarithmic

polynomial

exponential

Hierarchy of functions (3)

1

log n

n n2 n3 ... nk ...

2n

- Now, when we have a function, we can assign the function to some function in the hierarchy:
- For example, f(n) = 2n3 + 5n2 + 4n + 7The term with the highest power is 2n3.The growth rate of f(n) is dominated by n3.
- This concept is captured by Big-O notation

constant

logarithmic

polynomial

exponential

Big-O notation

- f(n) = O(g(n)) [read as f(n) is of order g(n)]
- Roughly speaking, this means f(n) is at most a constant times g(n) for all large n
- Examples
- 2n3 = O(n3)
- 3n2 = O(n2)
- 2n log n = O(n log n)
- n3 + n2 = O(n3)
- function on L.H.S and function on R.H.S are said to have the same order of magnitude

Big-O notation - formal definition

- f(n) = O(g(n))
- There exists a constant c and no such that f(n) cg(n) for all n > no
- cnon>no then f(n) cg(n)

Example – 2nlog2n+100n+1000 = O(n log n)

3 n log n

c=3, n0=600

f(n)

n log n

n0

constant c & no such that n>no, f(n) c g(n)

f(n) = O(g(n))?

c g(n)

f(n)

value of f(n) for n<n0does NOT matter

n0

constant c & no such that n>no, f(n) c g(n)

Which one is the fastest?

- Usually we are only interested in the asymptotic time complexity, i.e., when n is large
- O(log n) < O(n) < O(n) < O(n log n) < O(n2) < O(2n)

Proof of order of magnitude

- Show that 2n2 + 4n is O(n2)
- Since n n2 for all n>1,we have 2n2 + 4n 2n2 + 4n2 6n2for all n>1.
- Therefore, by definition, 2n2 + 4n is O(n2).

- Show that n3+n2logn+n is O(n3)
- Since n n3 and log n n for all n>1,we have n2 log n n3, andn3 + n2log n + n 3n3for all n>1.
- Therefore, by definition, n3+n2logn+n is O(n3).

Exercise

- Determine the order of magnitude of the following functions.
- n3 + 3n2 + 3
- 4n2 log n + n3 + 5n2 + n
- 2n2 + n2 log n
- 6n2 + 2n

O(n3)

O(n3)

O(n2 log n)

O(2n)

Exercise (2)

- n3 = n3 n
- 3n2 3n3 n1
- 3 3n3n1
- n3+3n2+3 7n3 n1

- Prove the order of magnitude
- n3 + 3n2 + 3
- n3 = n3 n
- 3n2 n3 n3
- 3 n3n2

n3 + 3n2 + 3 3n3 n3

- 4n2 log n + n3 + 5n2 + n
- 4n2 log n 4n3 n1
- n3 = n3 n
- 5n2 n3 n5
- nn3 n1

4n2 log n + n3 + 5n2 + n 7n3 n5

- 4n2 log n 4n3 n1
- n3 = n3 n
- 5n2 5n3 n1
- nn3 n1

4n2log n+n3+5n2+n 11n3 n1

Exercise (2) cont’d

- 2n2 2n2log n n2
- n2 log n = n2 log n n

2n2+n2log n 3n2log n n2

- Prove the order of magnitude
- 2n2 + n2 log n
- 2n2 n2 log n n4
- n2 log n = n2 log n n

2n2 + n2 log n 2n2 log n n4

- 6n2 + 2n
- 6n2 6*2n n1
- 2n = 2n n

6n2 + 2n 7*2n n1

More Exercise

- Are the followings correct?
- n2log n + n3 + 3n2 + 3 O(n2)?
- n + 1000 O(n)?
- 6n20 + 2n O(n20)?
- n3 + 5n2 logn + n O(n2 log n) ?

O(n3)

YES

O(2n)

O(n3)

Some algorithms we learnt

input n

sum = 0

for i = 1 to n do

begin

sum = sum + i

end

output sum

- Computing sum of the first n numbers

O(n)

O(1)

input n

sum = n*(n+1)/2

output sum

O(?)

O(?)

Finding the min value among n numbers

min = 0

for i = 1 to n-1 do

if (a[i] < a[min]) then

min = i

output a[min]

O(n)

O(?)

More exercise

- What is the time complexity of the following pseudo code?

for i = 1 to 2n do

for j = 1 to n do

x = x + 1

O(n2)

O(?)

The outer loop iterates for 2n times.

The inner loop iterates for n times for each i.

Total: 2n * n = 2n2.

Learning outcomes

- Able to carry out simple asymptotic analysis of algorithms
- See some examples of polynomial time and exponential time algorithms
- Able to apply searching/sorting algorithms and derive their time complexities

Searching

- Input: a sequence of n numbers a0, a1, …, an-1; and a number X
- Output: determine whether X is in the sequence or not
- Algorithm (Linear search):
- Starting from i=0, compare X with ai one by one as long as i < n.
- Stop and report "Found!" when X = ai .
- Repeat and report "Not Found!" when i >= n.

Linear Search

To find 7

- 12 34 2 9 7 5 six numbers7number X
- 12 34 2 9 7 57
- 12 34 2 9 7 57
- 12 34 2 9 7 57
- 12 34 2 9 7 57found!

Linear Search (2)

To find 10

- 12 34 2 9 7 510
- 12 34 2 9 7 510
- 12 34 2 9 7 510
- 12 34 2 9 7 510
- 12 34 2 9 7 510
- 12 34 2 9 7 510not found!

Linear Search (3)

- i = 0
- while i < n do
- begin
- if X == a[i] then
- report "Found!" and stop
- else
- i = i+1
- end
- report "Not Found!"

Time Complexity

i = 0

while i < n do

begin

if X == a[i] then

report "Found!" & stop

else

i = i+1

end

report "Not Found!"

- Important operation of searching: comparison
- How many comparisons this algorithm requires?

Best case: X is the 1st no., 1 comparison, O(1)

Worst case: X is the last no. OR X is not found, n comparisons, O(n)

Improve Time Complexity?

If the numbers are pre-sorted, then we can improve the time complexity of searching by binary search.

Binary Search

- more efficient way of searching when the sequence of numbers is pre-sorted
- Input: a sequence of n sorted numbers a0, a1, …, an-1 in ascending order and a number X
- Idea of algorithm:
- compare X with number in the middle
- then focus on only the first half or the second half (depend on whether X is smaller or greater than the middle number)
- reduce the amount of numbers to be searched by half

Binary Search (3)

To find 30

- 3 7 11 12 15 19 24 33 41 55 10 nos30X
- 19 24 33 41 5530
- 19 2430
- 2430not found!

Binary Search (4)

is the floor function,

truncate the decimal part

- first=0, last=n-1
- while (first <= last) dobegin
- mid = (first+last)/2
- if (X == a[mid]) report "Found!" & stop
- else if (X < a[mid])last = mid-1
- elsefirst = mid+1end
- report "Not Found!"

Time Complexity

Best case:X is the number in the middle=> 1 comparison, O(1)-time

Worst case:at most log2n+1 comparisons, O(log n)-time

Why? Every comparison reduces the amount of numbers by at least half

E.g., 16 => 8 => 4 => 2 => 1

first=0, last=n-1

while (first <= last) dobegin

mid = (first+last)/2

if (X == a[mid])

report "Found!" & stop

else if (X < a[mid]) last = mid-1

else

first = mid+1end

report "Not Found!"

Binary search vs Linear search

- Time complexity of linear search is O(n)
- Time complexity of binary search is O(log n)
- Therefore, binary search is more efficient than linear search

Search for a pattern

- We’ve seen how to search a number over a sequence of numbers
- What about searching a pattern of characters over some text?

Example

text:

N O B O D Y _ N O T I C E _ H I M

pattern:

N O T

substring:

N O B O D Y _ N O T I C E _ H I M

String Matching

- Given a string of n characters called the text and a string of m characters (mn) called the pattern.
- We want to determine if the text contains a substring matching the pattern.

Example

text:

N O B O D Y _ N O T I C E _ H I M

pattern:

N O T

substring:

N O B O D Y _ N O T I C E _ H I M

Example

T[ ]:

N O B O D Y _ N O T I C E _ H I M

N OT

P[ ]:

NO T

NO T

bolded: match

underlined: not match

un-bolded: not considered

NO T

NO T

NO T

NO T

N O T

The algorithm

- The algorithm scans over the text position by position.
- For each position i, it checks whether the pattern P[0..m-1] appears in T[i..i+m-1]
- If the pattern exists, then report found
- Else continue with the next position i+1
- If repeating until the end without success, report not found

Match pattern with T[i..i+m-1]

- j = 0
- while (j < m && P[j]==T[i+j]) do
- j = j + 1
- if (j == m) then
- report "found!" & stop

T[i] T[i+1] T[i+2] T[i+3] … T[i+m-1]

P[0] P[1] P[2] P[3] … P[m-1]

Match for all positions

- for i = 0 to n-m do
- begin
- // check if P[0..m-1] match with T[i..i+m-1]
- end
- report "Not found!"

Match for all positions

- for i = 0 to n-m do
- begin
- j = 0
- while (j < m && P[j]==T[i+j]) do
- j = j + 1
- if (j == m) then
- report "found!" & stop
- end
- report "Not found!"

Time Complexity

How many comparisons this algorithm requires?

Best case:pattern appears in the beginning of the text, O(m)-time

Worst case:pattern appears at the end of the text OR pattern does not exist, O(nm)-time

for i = 0 to n-m do

begin

j = 0

while j < m & P[j]==T[i+j] do

j = j + 1

if j == m then

report "found!" & stop

end

report "Not found!"

Sorting

- Input: a sequence of n numbers a0, a1, …, an-1
- Output: arrange the n numbers into ascending order, i.e., from smallest to largest
- Example: If the input contains 5 numbers 132, 56, 43, 200, 10, then the output should be 10, 43, 56, 132, 200

There are many sorting algorithms:insertion sort, selection sort, bubble sort, merge sort, quick sort

Selection Sort

- find minimum key from the input sequence
- delete it from input sequence
- append it to resulting sequence
- repeat until nothing left in input sequence

Selection Sort - Example

- sort (34, 10, 64, 51, 32, 21) in ascending order

Sorted part Unsorted part Swapped

34 10 64 51 32 21 10, 34

10 34 64 51 32 21 21, 34

10 21 64 51 32 34 32, 64

10 21 32 51 64 34 51, 34

10 21 32 34 64 51 51, 64

10 21 32 34 51 64 --

10 21 32 34 51 64

Selection Sort Algorithm

- for i = 0 to n-2 do
- begin
- // find the index of the minimum number
- // in the range a[i] to a[n-1]
- swap a[i] and a[min]
- end

Selection Sort Algorithm

how to swap two entries of an array?

using a temp variable?

not using a temp variable?

- for i = 0 to n-2 do
- begin
- min = i
- for j = i+1 to n-1 do
- if a[j] < a[min] then
- min = j
- swap a[i] and a[min]
- end

Algorithm Analysis

for i = 0 to n-2 do

begin

min = i

for j = i+1 to n-1 do

if a[j] < a[min] then

min = j

swap a[i] and a[min]

end

- The algorithm consists of a nested for-loop.
- For each iteration of the outer i-loop,there is an inner j-loop.

Total number of comparisons= (n-1) + (n-2) + … + 1= n(n-1)/2

O(n2)-time

Bubble Sort

- starting from the last element, swap adjacent items if they are not in ascending order
- when first item is reached, the first item is the smallest
- repeat the above steps for the remaining items to find the second smallest item, and so on

Bubble Sort - Example

- (34 10 64 51 32 21)round
- 34 10 64 51 32 21
- 1 34 10 64 51 21 32
- 34 10 64 21 51 32
- 3410 21 64 51 32don’t need to swap
- 34 10 21 64 51 32
- 10 34 21 64 51 32
- 2 10 34 21 64 32 51
- 10 3421 32 64 51 don’t need to swap
- 1034 21 32 64 51
- 10 21 34 32 64 51

underlined: being considered

italic: sorted

Bubble Sort - Example (2)

- round
- 10 21 34 32 64 51
- 3 10 21 3432 51 64don’t need to swap
- 10 2134 32 51 64
- 10 21323451 64don’t need to swap
- 4 10 213234 5164don’t need to swap
- 10 21323451 64don’t need to swap
- 5 10 21 32 34 5164

underlined: being considered

italic: sorted

Bubble Sort Algorithm

the smallest will be moved to a[i]

- for i = 0 to n-2 do
- for j = n-1 downto i+1 do
- if (a[j] < a[j-1])
- swap a[j] & a[j-1]

start from a[n-1], check up to a[i+1]

34 10 64 51 32 21

i = 0

j = 5

j = 4

j = 3

j = 2

j = 1

i = 1

j = 5

j = 4

j = 3

j = 2

Algorithm Analysis

- The algorithm consists of a nested for-loop.

for i = 0 to n-2 do

for j = n-1 downto i+1 do

if (a[j] < a[j-1])

swap a[j] & a[j-1]

Total number of comparisons= (n-1) + (n-2) + … + 1= n(n-1)/2

O(n2)-time

Sorting

- Input: a sequence of n numbers a0, a1, …, an-1
- Output: arrange the n numbers into ascending order, i.e., from smallest to largest

We have learnt these sorting algorithms:selection sort, bubble sort

Next: insertion sort (optional, self-study)

Insertion Sort (optional, self-study)

- look at elements one by one
- build up sorted list by inserting the element at the correct location

optional

Example

sort (34, 8, 64, 51, 32, 21) in ascending order

Sorted part Unsorted part int moved

34 8 64 51 32 21

34 8 64 51 32 21 -

8 34 64 51 32 21 34

8 34 64 51 32 21 -

8 34 51 64 32 21 64

8 32 34 51 64 21 34, 51, 64

8 21 32 34 51 64 32, 34, 51, 64

optional

Insertion Sort Algorithm

- for i = 1 to n-1 do
- begin key = a[i]
- pos = 0while (a[pos] < key) && (pos < i) do pos = pos + 1
- shift a[pos], …, a[i-1] to the right
- a[pos] = key
- end

using linear search to find the correct position for key

i.e., move a[i-1] to a[i], a[i-2] to a[i-1], …, a[pos] to a[pos+1]

finally, place key (the original a[i]) in a[pos]

optional

Algorithm Analysis

for i = 1 to n-1 do

begin key = a[i]

pos = 0while (a[pos] < key) && (pos < i) do pos = pos + 1

shift a[pos], …, a[i-1] to the right

a[pos] = key

end

- Worst case input
- input is sorted in descending order
- Then, for a[i]
- finding the position takes i comparisons

total number of comparisons= 1 + 2 + … + n-1= (n-1)n/2

O(n2)-time

optional

Selection, Bubble, Insertion Sort

- All three algorithms have time complexity O(n2) in the worst case.
- Are there any more efficient sorting algorithms? YES, we will learn them later.
- What is the time complexity of the fastest comparison-based sorting algorithm?O(n log n)

Traveling Salesman Problem

- Input: There are n cities.
- Output: Find the shortest tour from a particular city that visit each city exactly once before returning to the city where it started.

This is known as Hamiltonian circuit

Example

2

a

b

To find a Hamiltonian circuit from a to a

7

8

3

5

c

d

1

TourLength

a -> b -> c -> d -> a 2 + 8 + 1 + 7 = 18

a -> b -> d -> c -> a 2 + 3 + 1 + 5 = 11

a -> c -> b -> d -> a 5 + 8 + 3 + 7 = 23

a -> c -> d -> b -> a 5 + 1 + 3 + 2 = 11

a -> d -> b -> c -> a 7 + 3 + 8 + 5 = 23

a -> d -> c -> b -> a 7 + 1 + 8 + 2 = 18

Idea and Analysis

- A Hamiltonian circuit can be represented by a sequence of n+1 cities v1, v2, …, vn, v1, where the first and the last are the same, and all the others are distinct.
- Exhaustive search approach: Find all tours in this form, compute the tour length and find the shortest among them.

How many possible tours to consider?

(n-1)! = (n-1)(n-2)…1

N.B.: (n-1)! is exponential in terms of n

[ refer to batch 1 notes, slides #64,65 ]

Knapsack Problem

- Input: Given n items with weights w1, w2, …, wn and values v1, v2, …, vn, and a knapsack with capacity W.
- Output: Find the most valuable subset of items that can fit into the knapsack.
- Application: A transport plane is to deliver the most valuable set of items to a remote location without exceeding its capacity.

0 0

{1} 7 42

{2} 3 12

{3} 4 40

{4} 5 25

{1,2} 10 54

{1,3} 11 N/A

{1,4} 12 N/A

total totalsubsetweightvalue

{2,3} 7 52

{2,4} 8 37

{3,4} 9 65

{1,2,3} 14 N/A

{1,2,4} 15 N/A

{1,3,4} 16 N/A

{2,3,4} 12 N/A

{1,2,3,4} 19 N/A

Examplew = 7

v = 42

capacity = 10

w = 5

v = 25

w = 4

v = 40

w = 3

v = 12

item 1

item 2

item 3

item 4

knapsack

Idea and Analysis

- Exhaustive search approach: Try every subset of the set of n given items, compute total weight of each subset and compute total value of those subsets that do NOT exceed knapsack\'s capacity.

How many subsets to consider?

2n, why?

Exercises (1)

- Suppose you have forgotten a password with 5 characters. You only remember:
- the 5 characters are all distinct
- the 5 characters are B, D, M, P, Y
- If you want to try all possible combinations, how many of them in total?
- What if the 5 characters can be any of the 26 upper case alphabet?

5! = 120

26*25*24*23*22

or 26!/21!

Exercises (2)

- Suppose the password also contains 2 digits, i.e., 7 characters in total
- all characters are all distinct
- the 5 alphabets are B, D, M, P, Y
- the digit is either 0 or 1
- How many combinations are there?

7! = 5040

Exercises (3)

- What if the password is in the form adaaada?
- a means alphabet, d means digit
- all characters are all distinct
- the 5 alphabets are B, D, M, P, Y
- the digit is either 0 or 1
- How many combinations are there?

5! * 2! = 240

Learning outcomes

- Able to carry out simple asymptotic analysis of algorithms
- See some examples of polynomial time and exponential time algorithms
- Able to apply searching/sorting algorithms and derive their time complexities

Download Presentation

Connecting to Server..