1 / 81

Explorations in Artificial Intelligence

Explorations in Artificial Intelligence. Prof. Carla P. Gomes gomes@cs.cornell.edu Module 6 Intro to Complexity. The algorithm problem. Any legal input. Specification of all legal inputs. and. The algorithm. Specification of desired output as a function of the input. The desired

Download Presentation

Explorations in Artificial Intelligence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Explorations in Artificial Intelligence Prof. Carla P. Gomes gomes@cs.cornell.edu Module 6 Intro to Complexity

  2. The algorithm problem Any legal input Specification of all legal inputs and The algorithm Specification of desired output as a function of the input The desired output

  3. Problem 1: Input: A list L, of integers Output: The sum of the integers on L Problem 3: Input: A road map of cities with distances attached to the road map, and two designated cities A and B Output: A description of the shortest path between A and B Problem 2: Input: Two texts A and B in English Output: The list of common words in both texts Examples of algorithmic problems

  4. Examples of algorithmic problems Problem 5: Input: A valid linear programming problem Output: The optimal solution to the LP problem Problem 4: Input: A road map of cities with distances attached to the road map, and a designated initial city A. Output: A description of the shortest tour that passes through all the cities starting and ending at A.

  5. Variants of algorithmic problems: Decision Problem Problem:Knapsack (decision) Input: profits p0, p1, …, pn-1 weights w0, w1, …, wn-1 capacity M target profit P Output: YES or NO to the question: Does there exist an n-tuple [xo,…,x n-1] Є {0,1}n, such that: and

  6. Variants of algorithmic problems: Search Problem Problem:Knapsack (search) Input: profits p0, p1, …, pn-1 weights w0, w1, …, wn-1 capacityM target profit P Output: An n-tuple [xo,…,x n-1] Є {0,1}n, such that: and

  7. Variants of algorithmic problems: Optimal Value Problem:Knapsack (optimal value) Input: profits p0, p1, …, pn-1 weights w0, w1, …, wn-1 capacityM Output: The maximum value of: subject to: and [xo,…,x n-1] Є {0,1}n

  8. Variants of algorithmic problems: Optimization Problem:Knapsack (optimization) Input: profits p0, p1, …, pn-1 weights w0, w1, …, wn-1 capacity M Output: An n-tuple [xo,…,x n-1] Є {0,1}n, such that: is maximized subject to:

  9. Problem 1: Input: A list L, of integers Output: The sum of the integers on L Instance of an algorithmic problemSize of an instance • An instance of an algorithmic problem is a concrete case of such a problem with specific input. The size of an instance is given by the size of its input. • Examples of instances: • An instance of problem 1: • L= 2, 5, 26, 8, 170, 79, 1002 Size of instance  length of list Size of instance = |L| = 7

  10. 4 2 4 2 2 2 1 3 1 6 4 2 Problem 3: Input: A road map of cities with distances attached to the road map, and two designated cities A and B Output: A description of the shortest path between A and B 3 3 5 Examples of instances Size of instance  Number of cities and roads A particular instance: Size of instance: 6 nodes 9 edges The size of an instance is given by the size of its input.

  11. Problem 5: Input: A valid linear programming problem Output: The optimal solution to the LP problem The size of an instance is given by the size of its input. Size of instance  Number of variables and constraints (n,m) • An instance of problem 5: • max 3 x + 5 y • s.t • x ≤ 4 • y ≤ 12 • 3 x + 2 y ≤ 18 • x,y ≥ 0 Size of instance: 2 variables 3 functional constraints

  12. Complexity of Algorithms

  13. Complexity of Algorithms • The complexity of an algorithm is the number of steps that it takes to transform the input data into the desired output. • Each simple operation (+,-,*,/,=,if,etc) and each memory access corresponds to a step.(*) • The complexity of an algorithm is a function of the size of the input (or size of the instance). We’ll denote the complexity of algorithm A by CA(n), where n is the size of the input. (*) This model is a simplification but still valid to give us a good idea of the complexity of algorithms.

  14. Example: Insertion Sort From: Introduction to Algorithms Cormen et al

  15. Different notions of complexity Worst case complexity of an algorithm A – the maximum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. It provides an upper bound for an algorithm. The worst that can happen given the most difficult instance – the pessimistic view. Best case complexity of an algorithm A -the minimum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. The most optimisticview of an algorithm– it tells us the least work a particular algorithm could possibly get away with for some one input of a fixed size – we have the chance to pick the easiest input of a given size.

  16. Average case complexity of an algorithm A - i.e.,the average amount of resources the algorithm consumes assuming some plausible frequency of occurrence of each input. • Figuring out the average cost is much more difficult than figuring out either the worst-cost or best-cost  e.g., we have to assume a given probability distribution for the types of inputs we get.

  17. In general this is the notion that we use to characterize the complexity of algorithms Different notions of complexity We perform upper bound analysis on algorithms.

  18. Growth Rates • In general we only worry about growth rates because: • Our main objective is to analyze the cost performance of algorithms. • Another obstacle to having the exact cost of algorithms is that sometimes the algorithm are quite complicated to analyze. • When analyzing an algorithm we are not that interested in the exact time the algorithm takes to run – often we only want to compare two algorithms for the same problem – the thing that makes one algorithm more desirable than another is its growth rate relative to the other algorithm’s growth rate.

  19. Two functions of n have different growth rates if as n goes to infinity their ratioeither goes to infinity or goes to zero. If their ratio stays near a non-zero constantthen they are asymptotically the same function. Growth Rates

  20. Big Oh Notation • Given two functions F and G, whose domain is the natural numbers, we say that the order of F is lower than or equal to the order of G if: • F(n) ≤ c G(n) for all n > n0 (c and n0 are constants) • We say F is O(G) (F is oh of G) Example: (3 n3 + n2 + n ) is O(n3) In practice we just look at the fastest growing term of the expression

  21. Typical Growth Rates

  22. Roughly Speaking… exponential quadratic Cost linear logarithmic constant Size

  23. Good vs. Bad Algorithms

  24. How do computer scientists differentiate between • good (efficient) and bad (not efficient) algorithms? The yardstick is that any algorithm that runs inno morethanpolynomial time is an efficient algorithm; everything else is not.

  25. Ordered functions by their growth rates c Order constant 1 logarithmic 2 polylogarithmic 3 lg n nr ,0<r<1 n sublinear 4 lgc n linear 5 nr ,1<r<2 subquadratic 6 n2 quadratic 7 n3 cubic 8 nc,c≥1 rn, r>1 polynomial 9 Efficient algorithms exponential 10 Not efficient algorithms

  26. Polynomial vs. exponential growth (Harel 2000) Binary B&B alg and DPLL. exponential polynomial LP’s interior point Min. Cost Flow Algs Transportation Alg Assignment Alg Dijkstra’s alg. N2

  27. Problem Complexity Theory of NP-completeness or NP-hardness Easy vs. hard problems

  28. Overview of complexity • How can we show a problem is efficiently solvable? • We can show it constructively. We provide an algorithm and show that it solves the problem efficiently. E.g.: • Shortest path problem - Dijkstra’s algorithm runs in polynomial time, O(n2). Therefore the shortest path problem can be solved efficiently. • Linear Programming – The Interior Point method has polynomial worst-case complexity. Therefore Linear programming can be solved efficiently. (*) The simplex method has exponential worst case complexity/ However, in practice the simplex algorithm seems to scale as m3, where m is the number of functional constraints.

  29. Overview of complexity • How can we show a problem is not efficiently solvable? • How do you prove a negative? Much harder!!! • This is the aim of complexity theory.

  30. Easy (efficiently solvable) problems vsHard Problems • Easy Problems - we consider a problem X to be“easy” or efficiently solvable, if there is a polynomial time algorithm A for solving X. We denote by P the class of problems solvable in polynomial time. • Hard problems --- everything else. Any problem for which there is no polynomial time algorithm is an intractable problem. • .

  31. Problem 2: Input: A road map of cities with distances attached to the road map, and two designated cities A and B Output: A description of the longest path between A and B Problem 1: Input: A road map of cities with distances attached to the road map, and two designated cities A and B Output: A description of the shortest path between A and B We know that it is easy – it can be solved efficiently! Can we come up with an efficient algorithm for the longest path? Two problems

  32. I’d like you to develop an effcient algorithm to find the longest path between two points in a graph.

  33. 4 2 4 2 2 2 1 3 1 6 4 2 3 3 5 Your Longest Path Algorithm between two nodes, u and v G=(N,E) u v Initialization: MaxPath  none ; MaxPathLength  0; For each path P starting at 1: if P is a simple path from u to v and length(P) > MaxPath MaxPath  P; MaxPathLength  length(P) Return: MaxPath; MaxPathLength;

  34. I know…If each intersection leads to two roads and we have 100 intersections that can lead to 2100 paths … Is that the best you can do? -- that seems to be a bad algorithm!!!

  35. I cant’ find an efficient algorithm. I guess I’m too dumb.

  36. I can’t find an efficient algorithm, but neither can these famous researchers.

  37. Alan Turing In 1936, Alan Turing, a British mathematician, showed that there exists a relatively simple universal computing device that can perform any computational process. Computers use such a universal model. Turing Machine (abstraction) Turing also showed the limits of computation – some problems cannot be computed even with the most powerful computer and even with unlimited amount of time – e.g., Halting problem.

  38. John Von Neumann Brilliant mathematician, synthesizer, and promoter of the stored program concept, whose logical design of the Institute of Advanced Studies (Princeton) Computer became the prototype of today’s computer (*) - the von Neumann Architecture. (*) sequential i.e., non-parallel computers

  39. Richard Karp Steven Cook Invented the theory of NP-Completeness – proved that a simple problem - Satisfiability is NP-Complete. Given a propositional formula, is there an assignment to its variables (a, b, and c – True or False) making the formula true? Showed that several important problems and applications are NP-Complete and NP-hard, including Integer Programming.

  40. David Johnson Michael Garey

  41. George Dantzig • Invented Linear Programming Formulations: • max 3 x + 5 y • s.t • x ≤ 4 • y ≤ 12 • 3 x + 2 y ≤ 18 • x,y ≥ 0 • Invented Simplex Algorithm

  42. Theory of NP-completeness and NP-hardnessEasy vs. hard problems

  43. Can satisfiability or integer programming be solved in polynomial time? • FACT: every algorithm that has ever been developed for satisfiability or integer programming takes exponential time. • Hundreds of very smart researchers have tried to come up with polynomial time algorithms for satisfiability or integer programming, and failed. • It is generally believed that there is no polynomial time algorithm for satisifiability or integer programming. • Complexity theory: deals with proving that satisfiability, integer programming, and many other problems are computationally hard.

  44. Given a propositional formula, is there an assignment of values True or False to its variables that makes the formula true? Decision Problems NP-Completeness theory deals with decision problems. What is a decision problem? A problem for which there is a yes/no answer. Examples: Is there a path between two nodes in a graph shorter than k? (decision version of shortest path problem) Is there a path between two nodes in a graph longer than k? (decision version of longest path problem) Most optimization problems can be formulated as a decision problem

  45. NP P Class NP Class of Problems solvable in Nondeterministic-Polynomial Time • We say that a decision problem is solvable in Non-deterministic polynomial time if: • The solution can be verified in polynomial time. (E.g., verifying that a path has length greater than K) • If we imagine that we have an exponential number of processors, we can check all possible solutions simultaneously and therefore answer in polynomial time

  46. NP NP-Complete P Class NP-Complete • The first problem to be shown to be NP-Complete was Satisfiability – Cook showed that all the problems in NP could be translated (in polynomial time) as Satisfiability problems; • The word complete means that every problem in the class NP-complete can be reduced (in polynomial time) into another problem of the class NP-complete. For example all the problems in the class NP-complete can be written as Satisfiability problems. • The class of NP-Complete problems is the class of the hardest computational problems in the class NP: every NP problem can be transformed into an NP-complete problem (the reverse is not true!!!)

  47. Is P not equal to NP?$1,000,000 question • P not equal to NP? • Is that true that not all problems is NP can be solved in polynomial time? • Class of NP-Complete Problems • They all admit exponential time solutions; • Nobody has ever been able to find a polynomial time solution for any single problem in the class; • Nobody has ever been able to prove an exponential lower bound for any single problem in the class;

  48. NP NP-Complete P P not equal to NP?$1,000,000 question • Pictorial interpretation of this question Is this the right picture? There are problems in NP that are inherently intractable and cannot be solved in polynomial time.

  49. P not equal to NP?$1,000,000 question • Pictorial interpretation of this question Or is this the right picture? All the problems in NP can be solved in polynomial time. Even though at this point we don’t know of polynomial time algorithms to solve some problems in NP, they exist… P = NP

  50. Class of NP-Complete ProblemsOne Million Dollar Prize • Completeness • – if someone were to find a polynomial time solution for a single problem in the class NP-complete  all the problems could be solved in polynomial time!!! • if someone were to prove an exponential lower bound for a single problem in the class NP-complete  all the problems in the class would be intractable !!!

More Related