Computability and Computational Complexity

1. Computability and Computational Complexity Undecidable (no algorithms at all) Problems Provably intractable (exponential time) Presumably intractable Tractable (polynomial time) Decidable

2. Problems described by: • a general description of all its parameters • a statement of what properties the solution is required to satisfy • an instance of a problem is obtained by specifying particular values for all problem parameters

3. Example: the Traveling Salesman Problem (TSP) C = {c1, c2, ..., cK} cities D = {d(c1, c2), d(c1, c3),..., d(cK-1, cK)} distance between them Find <c(1), c(2), ..., c(K)> that minimizes K-1  [ d(c(j), c(j+1)) ] + d(c(1), c(K)) j=1

4. c1 C = {c1, c2, c3 ,c4} d(c1, c2) = 10 d(c1, c3) = 5 d(c1, c4) = 9 d(c2, c3) = 6 d(c2, c4) = 9 d(c3, c4) = 3 5 10 9 c3 6 3 c4 c2 9 <c1, c2, c4 ,c3> is the solution 10+9+3+5 whose length is 27

5. Instance description: a string that describes the instance for exaple, using alphabet {c,[,],/,0,1,2,3,4,5,6,7,8,9} we can describe the preceding instance by the following string c[1] c[2] c[3] c[4] // 10/5/9 // 6/9 // 3 whose length is the size of the input

6. Algorithm step by step procedure to solve the problem in practice: a computer program written in a specific language that always find the solution for example, in the TSP it always constructs an ordering that gives the minimum length tour for each instance In general we ask for efficient algorithms, which minimizes in some sense the resources (time or space) needed to solve a problem

7. Time maximum number of steps we need to solve Complexity an instance of the problem as function of the size of the input Inputsize length of the instance description obtained from the encoding scheme used

8. Tractable and intractable problems (polynomial vs exponential) We say that a function f(n) is O(g(n)) whenever there exists a constant c such that |f(n)| ≤ c |g(n)| for all n ≥ 0 Polynomal time algorithm: O(p(n)) for some polynomial function p (n input length)

9. Any algorithm whose time complexity function cannot be bounded by some polynomial function is said to be an exponential time algorithm Examples O(2n) O(nn) O(n!) but also O(nlog n) O(nlog log n) O(nsqrt(n))

10. Examples of polynomal time algorithms

11. Most exponential algorithms are merely variations on exhaustive search Polynomial time algorithms (PTA) are generally made possible only through the gain of some deeper insight into the structure of a problem In a sense, a problem is not well solved util a polynomial time algorithm is found for it A problem is intractable if it is so hard that no polynomial time algorithm can possibly solve it

13. pol exp computer with 106 op/s

14. pol exp

17. Remarks • Tractable vs intractable can be applied only when the size of the problem is not too limited Example: in figure 1.2 2n algorithm is faster than n5 when n ≤ 22 molto importante! • Time complexity is defined as a worst-case measure: at least one problem instance of size n requires that much time; but in some cases most problem instances require far less time (Cryptography)

18. Remarks • even if algorithms with complexityt n100 or 1099 n2 cannot be considered efficient in practice, the vast majority of polynomial time algorithms used to solve natural problems have a degree that is 2 or 3 at worst, and do not involve large constant c

19. Natural questions: • What encoding used to describe instances? • What model of computation? The intractability of a problem turns out to be essentially independent of the particular encoding scheme and computer model used

20. Possible encoding schemes Use the case of a graph G = (V, E) as an example (V the set of vertices, E the set of edges) • [Vertex-edge list] - list all vertices and edges • [Neighbor list] - list, for each vertex, all the other vertices sharing a common edge with it (neighbor list) 3. [Adjacency matrix rows] - list the rows of the adjacent matrix for the graph

21. 1 2 3 1 2 3 4 1 2 3 4 V = {V[1], V[2], V[3], V[4]} E = {{V[1], V[2]}, {V[2], V[3]}} V[2] V[4] V[1] V[3]

22. General bounds on input lengths for the three previous encoding schemes v = |V| e = |E| Remember that e ≤ v (v -1)/2 and so the input lengths differs at most polynomially from each other

23. Model of computation Time required by Machine A to simulate execution of an algorithm of time complexity T(n) on Machine B

24. Why to study complexity theory? • if a problem is inherently intractable, proving this could be just as hard as finding efficient algorithms • complexity theory provides techniques to prove that a problem is just as hard as other problems known to be difficult (reduction) then, as a consequence...

25. one might look for efficient algorithms to solve special cases of the probem or • one might search for algorithms that run quickly most of the time In other words complexity theory is used to assist algorithm designers in directing their problem-solving efforts towards those approach that have greatest likelihood to leading to useful algorithms

26. Computational Complexity Undecidable (no algorithms at all) Problems Provably intractable (exponential time) Presumably intractable Tractable (polynomial time) Decidable

27. but also Examples of Undecidable (unresolvable at all) • the Halting problem on a Turing Machine (already studied) • undecidable results from Rice theorem (such as x= y) • validity of the first order logical calculus (Entscheidungproblem) • tiling a floor • domino snakes • word correspondence of Post • ... many others

28. 1) The Halting problem Given a TM (a RAM machine, a Post machine, ...), i.e. given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input. Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist. We say that the halting problem is undecidable.

29. 2) The Tiling Problem Given a set of tiles, decide if any finite area, of any size, can be covered such that colors of any adjacent edges are the same

31. Furthermore, very strange things can happen: 3) Domino snake • finite portion -> decidable • infinite plane -> decidable • semi-infinite plane -> undecidable!!

32. 4) Word correspondence of Post Admits correspondence: 2,1,1,4,1,5 a|a b b|a b b|b a b a|a b b|a b a a a|b b a b|b b a b|a a|b b a b|a Admits no correspondence Post correspondence is undecidable!

33. The following slightly different problem is, on the contrary, decidable: the words from Y need not to be in correspondence 3 2 2 b a b|a|a b a b|a a 1 2

34. Examples of Provably Intractable problems (surely unresolvable in practice) 1) The Equivalence Problem for Regular Expressions with Squaring Requires Exponential Space by Albert Meyer and Larry Stockmeyer, FOCS 1972. Regular Expressions •  (empty string) and  (empty set) are regular expressions • Given s , s is a regular expression • If R1 and R2 are regular expressions, then also (R1+R2), (R1  R2) and (R1)* are regular expressions union concatenation

35. For any finite set  of symbols, * (Kleene star) is the set of all finite strings of symbols from . examples: (a + b)* all words a*ba*ba all words with exactly two b (a + b)*(aa + bb)*(a + b) * all words containing two consecutive a or b a aa aaa aaaa ...

36. The title result of this paper gave an early example of a natural problem that provably does not have an efficient algorithm. This is the problem of recognizing whether two regular expression represent different languages, where the expressions are limited to four operators: union, concatenation, the Kleene star, and squaring (two copies of an expression)

37. 2) Another more intuitive example: The Hanoi tower Move the entire stack to another rod, obeying the following rules: • Only one disk may be moved at a time. • Each move consists of taking the upper disk from one of the rods and sliding it onto another rod, on top of the other disks that may already be present on that rod. 3) No disk may be placed on top of a smaller disk. 2n -1 moves with n disks

38. the problem is so difficult that an exponential time algorithm is necessary to discover a solution • the solutionitself is required to be so extensive that it cannot be described with an expression having length bounded by a polynomial function Usually we have two causes for intractability:

39. Example for case 1: For an n*n chessboard with an arrangement of pieces, determine whether there is a winning strategy for White (i.e. a sequence of moves so that, no matter what Black does, White is guaranteed to win). Example for case 2: • Hanoi Tower • variant of the TSP with a parameter B for which we ask for all tours with a length less than B. An exponential number of tours may occur in some situations

40. Presumably intractable problems

41. Foundations of NP-Completeness were laid by Sthephen Cook in 1971, with the paper entitled “The Complexity of Theorem Proving Procedures” Outlined key concepts: • polynomial time reducibility • the class NP of decision problems that can be solved by a nondeterministic computer (Nondeterministic Polynomial Time) • one particular problem in NP, called the “satisfiability” problem (SAT), has the property that any other problem in NP can be reduced to it • other problems in NP might share the property of being the “hardest” member of NP (for example: does a given graph contain a complete subgraph on a given number kof vertices?)

42. 1) Polynomial time reducibility: If we have a polynomial time reduction from one problem to another, this ensures that any polynomial time algorithm for the second problem can be converted into a corresponding polynomial time algorithm for the first problem 2) class NP of decision problems that can be solved by a nondeterministic computer A nondeterministic computer performs unbounded parallel computation (many different actions may apply for the same combination of state and symbol) Remarks

43. 4) Karp proved, in 1972, that a large collection of well known combi- natorial problems (such as TSP) are just as hard as SAT. This collection of problems has been called the class of NP-complete problems 3) If the satisfiability problem can be solved with a polynomial time algorithm, then so can every other problem in NP. If any problem in NP is intractable, then SAT also must be intractable SAT, in a sense, is the hardest problem

44. Crucial question: are the NP-complete problems intractable? One of the foremost open questions of contemporary mathematics and computer science (One million dollars prize !!)

45. Theory of NP-completeness To begin we need to give formal counterparts for many informal notions such as “algorithm”, “problem” and so on. As a first step we need to use Decision problems that is problems with a yes/no answer

46. Examples of decision problems Subgraph isomorphism Instance: Two graphs G1 = (V1, E1) and G2 = (V2, E2) Question: Does G1 contain a subgraph isomorphic to G2?

47. K-1  [ d(c(j), c(j+1)) ] + d(c(1), c(K)) ≤ B j=1 Traveling salesman problem Instance: a set C = {c1, c2, ..., cK} of cities and a set D = {d(c1, c2), d(c1, c3),..., d(cK-1, cK)} of distances between them, and a bound B Question: is there a tour of all the cities having total length no more than B? That is an ordering of <c(1), c(2), ..., c(K)> such that

48. Subset sum problem Instance: a set S = {i1, i2, ..., ik} of integers Question: does the sum of some non-empty subsets equal exactly zero? Example: {3, -4, 7, -8, 8, -11, 29, 19, 24, -30, 33} 3 + 7 + 19 + 24 -11 - 8 - 4 - 30 = 0

49. For example, if one can find a minimum length solution in polynomial time for the TSP, then one can compare it with the bound B, and so one can solve in polynomial time the associated decision problem On the other hand, if one can prove that TSP (in the decision problem approach) is NP-complete, we would know that the TSP optimization problem is at least as hard. Moreover, decision problems are frequently no easier than the corresponding optimization problem Why decision problems? They are strictly connected with the corresponding optimization problem

50. Decision problems have a natural formal counterpart, that can be studied in a mathematically precise theory of computation, that is Languages Language For any finite set  of symbols, let * be the set of all finite strings of symbols from . If L is subset of *, we say that L is language over .