Algorithm Design and Analysis (ADA)

1. Algorithm Design and Analysis (ADA) 242-535, Semester 1 2013-2014 • Objective • to give some background on the course Please ask questions 0. Preliminaries Who I am: Andrew DavisonWiG Lab ad@fivedots.coe.psu.ac.th

2. Overview 1. What is an Algorithm? 2. Meeting Times / Locations 3. Workload 4. Exercises 5. Course Materials 6. Books • Videos • Web Sites

3. 1. What is a Algorithm? • An algorithm is a finite set of unambiguous instructions for solving a problem. • An algorithm is correct if on all legitimate inputs, it outputs the right answer in a finite amount of time • Can be expressed as • pseudocode • flow charts • text in a natural language (e.g. English) • computer code

4. Algorithm Design The theoretical study of how to solve computational problems • sorting a list of numbers • finding a shortest route on a map • scheduling when to work on homework • answering web search queries • and so on...

5. The Importance of Algorithms • Their impact is broad and far-reaching. • Internet. Web search, packet routing, distributed file sharing, ... • Biology. Human genome project, protein folding, ... • Computers. Circuit layout, file system, compilers, ... • Computer graphics. Movies, video games, virtual reality, ... • Security. Cell phones, e-commerce, voting machines, ... • Multimedia. MP3, JPG, DivX, HDTV, face recognition, ... • Social networks. Recommendations, news feeds, advertisements, ... • Physics. N-body simulation, particle collision simulation, ...

6. The Top 10 Algorithms of the 20th Century • Ten algorithms having "the greatest influence on the development and practice of science and engineering in the 20th century". • Dongarra and SullivanTop Ten Algorithms of the CenturyComputing in Science and EngineeringJanuary/February 2000 • Barry CipraThe Best of the 20th Century: Editors Name Top 10 AlgorithmsSIAM NewsVolume 33, Number 4, May 2000 • http://www.siam.org/pdf/news/637.pdf

7. What are the Top 10? • 1946: The Metropolis (Monte Carlo) Algorithm. Uses random processes to find answers to problems that are too complicated to solve exactly. • 1947: Simplex Method for Linear Programming.A fast technique for maximizing or minimizing a linear function of several variables, applicable to planning and decision-making. • 1950: Krylov Subspace Iteration Method.A technique for rapidly solving the linear equations that are common in scientific computation.

8. 1951: The Decompositional Approach to Matrix Computations. A collection of techniques for numerical linear algebra. • 1957: The Fortran Optimizing Compiler. • 1959: QR Algorithm for Computing Eigenvalues. A crucial matrix operation made swift and practical. Application areas include computer vision, vibration analysis, data analysis. • 1962: Quicksort Algorithm. We will look at this.

9. 1965: Fast Fourier Transform (FFT). It breaks down waveforms (like sound) into periodic components. Used in many different areas (e.g. digital signal processing , solving partial differential equations, fast multiplication of large integers.) • 1977: Integer Relation Detection. A fast method for finding simple equations that explain collections of data. • 1987: Fast Multipole Method. Deals with the complexity of n-body calculations. It is applied in problems ranging from celestial mechanics to protein folding.

10. Some Algorithm Types • Simple recursive algorithms • Divide and conquer • Backtracking • Dynamic programming • Greedy algorithms • Brute force • Randomized algorithms

11. Simple Recursive • A simple recursivealgorithm: • Non-recursive base case • Recurs with a simpler subproblem • Examples: • Count the number of occurrence of an element in a tree • Test if a value occurs in a list • Fibonnaci number calculation • Several of the other algorithm types use recursion in more complex ways

12. Fibonnaci numbers • The Fibonacci sequence: • 1, 1, 2, 3, 5, 8, 13, 21, 33, ... • Find the nth Fibonacci number: fib(int n) if (n <= 1) return 1; else return fib(n-1) + fib(n-2);

13. lots of repeated work, that only gets worse for bigger n • Execution of fib(5):

14. Divide and Conquer • A divide and conquer algorithm: • Divide the problem into smaller subproblems • Combine the solutions to the subproblems into a solution to the original problem • Examples: • quicksort • merge sort • binary search

15. Backtracking • Backtracking algorithms use a depth-first recursive search • involves choice (non-determinism) • backtracking means "go back to where you came from" • Examples: • search a maze • color a map with no more than four colors

16. Dynamic Programming • A dynamic programming algorithm remembers past results and uses them to find new results. • multiple solutions exist; find the “best” one (the optimal one) • the problen contains overlapping (repeated) subproblems • solutions are stored and reused

17. Fibonacci numbers Again • Finding the nth Fibonacci number involves lots of repeated work (overlapping subproblems) that can be avoided using memorization:

18. Greedy algorithms • An optimization problem: find the best solution • A “greedy algorithm” sometimes works well for optimization problems, and is easy to code • At each step: • use the best solution you can get right now, without regard for future steps • You hope that by choosing a localoptimum at each step, you will end up with a globallyoptimum final solution

19. Example: Counting Money • Count out some money using the fewest possible notes and coins • A greedy algorithm will take the largest possible note or coin at each step • Example: count out $6.39 using: • a $5 bill • a $1 bill // to make $6 • a 25¢ coin // to make $6.25 • a 10¢ coin // to make $6.35 • four 1¢ coins // to make $6.39

20. Greedy Algorithms Can 'Fail' • “Krons” money come in 1, 7, and 10 coins • Count out 15 krons: • A 10 kron piece • Five 1 kron pieces, for a total of 15 krons • This requires 6 coins, but a better solution is two 7 kron pieces and one 1 kron piece (3 coins) • The greedy algorithm 'fails' because its final solution is not the best (not globally optimal)

21. Brute Force • A brute force algorithm tries all possibilities until a satisfactory solution is found. • Often, brute force algorithms require exponential running time (very large time, so very slow) • Various heuristics and optimizations can be used • heuristic means “a rule of thumb” • look for sub-optimal solutions (not the best), which can be calculated more quickly than the best one

22. Randomized Algorithms • A randomized algorithm uses a random number to make a choice during the computation • faster than calculating a choice • the choice may be just as good • Examples: • Quicksort randomly chooses a pivot • Factor a large number by choosing random numbers as possible divisors

23. Analysis of Algorithms The theoretical study of algorithm performanceand resource usage. Performance isn't the only important things for code: • modularity • user-friendliness • correctness • programmer time • maintainability • simplicity • functionality • extensibility • robustness • reliability This subject isn't about these things.

24. Why study Algorithm Analysis? • It help us to understand algorithm scalability. • Performance often draws the line between what is feasible and what is impossible. • Algorithmic mathematics provides a precise way totalk about program behavior. • The lessons of program performance generalize to other computing resources.

25. Course Structure • Mathematical induction, running time of programs; growth of functions • Divide-and-conquer; comparison and linear sorts • Dynamic programming • Greedy algorithms • Elementary graph algorithms, minimum spanning trees, shortest path problems, maximum flow • String matching • Computational geometry • NP completeness; approximation algorithms

26. 2. Meeting Times / Locations • Tuesday 9:00 – 10:30 R101Wednesday10:30 – 12:00 R101 • I want to change these times to be 3 classes/week, each of 1 hour. • Tell me your preferences.

27. 3. Workload • Mid-term exam: 35% (2 hours) • week 8 • Final exam: 45% (3 hours) • weeks 17-18 • Two exercises: 20%(2*10) • weeks 6-7 (July 8-19) and weeks 15-16 (September 9-20)

28. Non-Attendence Penalty • I may take registration at the start of a class. • If someone is not there, they lose 1%(unless they have a good excuse). • A maximum of 10% can be lost • deducted from your final mark

29. 4. Exercises • The two exercises are worth a total of 20% (each worth 10%). • They will be maths problems and/or algorithms to design/write. continued

30. Planned exercise times (which may change): • ex. 1 in weeks 6-7 (July 8-19) • ex. 2 in weeks 15-16 (September 9-20) • Cheating will result in 0 marks. • YOU HAVE BEEN WARNED!!

31. 5. Course Materials • All the handouts (and other materials)will be placed on-line at http://fivedots.coe.psu.ac.th/Software.coe/242-535_ADA/ • Print 6 slides-per-page, grayscale, and bring to class.

32. 6. Books • Introduction to AlgorithmsThomas Cormen, Charles Leiserson, Ronald Rivest, Clifford SteinMcGraw Hill, 2003, 2nd edition • mathematical, advanced, the standard text • now up to version 3 (MIT) • lots of resources online; see video section continued

33. AlgorithmsRobert Sedgewick, Kevin WayneAddison-Wesley, 2011, 4th ed. • implementation (Java) and theory • intermediate level • Data Structures and Algorithms in JavaRobert LaforeSams Publishing, 2002, 2nd ed. • Java examples; old • basic level; not much analysis

34. Fun Overviews • Algorithms UnlockedThomas H. CormenMIT Press, March 2013 • Nine Algorithms that Changed the FutureJohn MacCormickPrinceton University Press, 2011 • http://users.dickinson.edu/~jmac/9algorithms/ • search engine indexing, pagerank, public key cryptography, error-correcting codes, pattern recognition, data compression, databases, digital signatures, computablity

35. Algorithmic PuzzlesAnany Levitin, Maria LevitinOxford University Press, , 2011 • Algorithmics: The Spirit of ComputingDavid Harel, Yishai FeldmanAddison-Wesley; 3 ed., 2004(and Springer, 2012) • http://www.wisdom.weizmann.ac.il/~harel/algorithmics.html

36. The New Turing Omnibus: Sixty-Six Excursions in Computer ScienceA. K. DewdneyHolt, 1993 • 66 short article; e.g. detecting primes, noncomputable functions, self-replicating computers, fractals, genetic algorithms, Newton-Raphson Method, viruses

37. 7. Videos • MIT 6.046J / 18.410J Intro. to Algorithms, Fall 2005 • http://ocw.mit.edu/6-046JF05 • original course website for Cormen book • http://videolectures.net/mit6046jf05_introduction_algorithms/ • video and slides side-by-side • http://www.catonmat.net/category/introduction-to-algorithms • notes taken while watching the videos

38. Hi-tech TrekRoyal Institution Christmas Lectures 2008 • A hi-tech trek through the world of computer science by Professor Chris Bishop; aimed at school kids. • Lecture 1: Breaking the speed limit • Lecture 2: Chips with everything • Lecture 3: Ghost in the machine • Lecture 4: Untangling the web • Lecture 5: Digital intelligence • http://richannel.org/christmas-lectures/2008/ • I have copies on a DVD

39. 8. Web Sites • Algorithm Tutorials • http://community.topcoder.com/tc?module=Static&d1=tutorials&d2=alg_index • Algorithmy • http://en.algoritmy.net/ • brief explanations and code • Algorithmist • http://algorithmist.com/index.php/Main_Page • explanations and code • Wikipaedia page for an algorithm • e.g. http://en.wikipedia.org/wiki/Quicksort