1 / 63

Linear Programming

Thinking about Algorithms Abstractly. Linear Programming. Def and Hot Dog Example Network Flow Def n Matrix View of Linear Programming Hill Climbing Simplex Method Dual Solution Witness to Optimality Define Dual Problem Buy Fruit or Sell Vitamines Duality Primal-Dual Hill Climbing.

kael
Download Presentation

Linear Programming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Thinking about Algorithms Abstractly Linear Programming • Def and Hot Dog Example • Network Flow Defn • Matrix View of Linear Programming • Hill Climbing Simplex Method • Dual Solution Witness to Optimality • Define Dual Problem • Buy Fruit or Sell Vitamines Duality • Primal-Dual Hill Climbing Jeff Edmonds York University Lecture5 COSC 3101

  2. Linear Programming Applied in various industrial fields: Manufacturing, Supply-Chain, Logistics, Marketing… To save money and increase profits ! • Linear Program: An optimization problem whose constraints and cost function are linear functions • Goal: Find a solution which optimizes the cost. E.g. Maximize Cost Function : 21x1 - 6x2 – 100x3 - 100x4 Constraint Functions: 5x1 + 2x2 +31x3 - 20x4 21 1x1 - 4x2 +3x3 + 10x1³ 56 6x1 + 60x2 - 31x3 - 15x4 200 …..

  3. A Hotdog A combination of pork, grain, and sawdust, …

  4. Constraints: • Amount of moisture • Amount of protein, • …

  5. The Hotdog Problem Given today’s prices,what is a fast algorithm to find the cheapest hotdog?

  6. Abstract Out Essential Details sawdust grain water pork Amount to add: x1, x2, x3, x4 Cost of Hotdog: 29x1 + 8x2 + 1x3 + 2x4 3x1 + 4x2 – 7x3 + 8x4 ³ 12 • Constraints: • moisture • protein, • … 2x1 - 8x2 + 4x3 - 3x4 ³ 24 -8x1 + 2x2 – 3x3 - 9x4 ³ 8 x1 + 2x2 + 9x3 - 3x4 ³ 31 Cost: 29, 8, 1, 2

  7. Abstract Out Essential Details Minimize: 29x1 + 8x2 + 1x3 + 2x4 Subject to: 3x1 + 4x2 – 7x3 + 8x4 ³ 12 2x1 - 8x2 + 4x3 - 3x4 ³ 24 -8x1 + 2x2 – 3x3 - 9x4 ³ 8 x1 + 2x2 + 9x3 - 3x4 ³ 31

  8. Network Flow as a Linear Program • Given an instance of Network Flow: <G,c<u,v>>express it as a Linear Program: • The variables: • Maximize: • Subject to: Flows f<u,v>for each edge. rate(F) = u F<u,t> - v F<t,v> <u,v>:F<u,v>c<u,v>. (Flow can't exceed capacity) v: u F<u,v> = wF<v,w> (flow in = flow out)

  9. Linear Programming minimize subject to Cj Xj Mi,j Ni Xj ³ Linear Program Minimize: CTX Subject to: MX³ N ³ Xj 0 • n variable xj that we are looking for values of. • An optimization function • Each has a variable has a coefficient cj. • The dot product CTX gives one value to minimize. • m constraints: • Some linear combination of the variablesmust be at least some set value. • i MiX ³ Ni • Generally implied that variables are positive.

  10. Linear Programming maximize minimize subject to subject to Cj Cj Mi,j Mi,j Ni Ni Xj Xj Xj  ³  Linear Program Minimize: CTX Subject to: MX³ N = Maximize: CTX Subject to: MX³ N Xj ³ = • These are the linear programs in “standard” form • Minimize CTX hence X subject to X ³ • Maximize CTX hence X subject to X  • But you could mix and match.

  11. Simplex Algorithm • Invented by George Dantzig in 1947 • A hill climbing algorithm Global Max Local Max

  12. Simplex Algorithm • Invented by George Dantzig in 1947 • A hill climbing algorithm • Guaranteed to find an global optimal solution for Linear Programs Global Max

  13. Simplex Algorithm • Computes solution to a Linear Program by evaluating vertices where constraints intersect each other. • Worst case exponential time. • Practically very fast. • Ellisoid algorithm (1979)First poly time O(n4L)algorithm.

  14. With n variables, x1, x2, … , xn, there are n dimensions. (Here n=2) • Each constraint is an n-1 dimensional plain. (Here the 1-dim line) • Each point in this space, is a solution. • It is a valid solution if it is on the correct side of each constraint plain Simplex Algorithm x2 Minimize Cost = 5x1 + 7x2 Constraint Functions C1: 2x1 + 4x2³ 100 C2: 3x1 + 3x2³ 90 x1, x2³ 0 C1 C2 (0,0) x1

  15. Solutions on this line have one value of the objective function • This has another • These are not valid • This is the optimal value Simplex Algorithm x2 Cost Function Cost = 5x1 + 7x2 Constraint Functions C1: 2x1 + 4x2³ 100 C2: 3x1 + 3x2³ 90 x1, x2³ 0 C1 C2 (0,0) x1

  16. The arrow tells the direction that the optimal function increases. • Note that the solution is a vertex (simplex). • Each simplex is the intersection of n constraints. (Here n = #of variables = 2) Simplex Algorithm x2 Cost Function Cost = 5x1 + 7x2 Constraint Functions C1: 2x1 + 4x2³ 100 C2: 3x1 + 3x2³ 90 x1, x2³ 0 C1 C2 (0,0) x1

  17. The arrow tells the direction that the optimal function increases. • Note that the solution is a vertex (simplex). • Each simplex is the intersection of n constraints. (Here n = #of variables = 2) • The simplex method takes hill climbing steps, from one simplex (valid solution) to another. Simplex Algorithm

  18. Simplex Algorithm • With n variables, x1, x2, x3 … , xn, there are n dimensions. (Here n=3) • Each constraint is an n-1 dimensional plain. (Here the 2-dim triangles) • Each simplex (vertex) is the intersection of n such constraints. (Here looks like 6 but generally only n=3) A simplex is specifiedby a subset of n of the m tight (=) constraints. Maximize Cost : 21x1 - 6x2 – 100x3 Constraint Functions:5x1 + 2x2 +31x3211x1 - 4x2 +3x3 566x1 + 60x2 - 31x3 200 ⁞-5x1 + 3x2 +4x3 8 ⁞x1, x2, x3 ³ 0 All other constraints must be satisfied.

  19. Simplex Algorithm • If we slacken one of our n tight constrains, our solution slides along a 1-dim edge. • Head in the direction that increases the potential function. A simplex is specifiedby a subset of n of the m tight (=) constraints. Maximize Cost : 21x1 - 6x2 – 100x3 Constraint Functions:5x1 + 2x2 +31x3 211x1 - 4x2 +3x3  566x1 + 60x2 - 31x3  200 ⁞-5x1 + 3x2 +4x3  8 ⁞x1, x2, x3 ³ 0

  20. Simplex Algorithm • If we slacken one of our n tight constrains, our solution slides along a 1-dim edge. • Head in the direction that increases the potential function. • Keep sliding until we tighten some constraint. • This is one hill climbing step. A simplex is specifiedby a subset of n of the m tight (=) constraints. Maximize Cost : 21x1 - 6x2 – 100x3 Constraint Functions:5x1 + 2x2 +31x3 211x1 - 4x2 +3x3  566x1 + 60x2 - 31x3  200 ⁞-5x1 + 3x2 +4x3  8 ⁞x1, x2, x3 ³ 0

  21. Simplex Algorithm • If we slacken one of our n tight constrains, our solution slides along a 1-dim edge. • Head in the direction that increases the potential function. • Keep sliding until we tighten some constraint. • This is one hill climbing step. A simplex is specifiedby a subset of n of the m tight (=) constraints. Maximize Cost : 21x1 - 6x2 – 100x3 Constraint Functions:5x1 + 2x2 +31x3 211x1 - 4x2 +3x3  566x1 + 60x2 - 31x3  200 ⁞-5x1 + 3x2 +4x3  8 ⁞x1, x2, x3 ³ 0

  22. Simplex Algorithm • Do all Linear Programs have optimal solutions ? No ! • Three types of Linear Programs: 1. Has an optimal solution with a finite cost value: e.g. nutrition problem 2. Unbounded: e.g maximize x, x ³ 5, x ³ 0 3. Infeasible: e.g maximize x, x  3, x ³ 5 , x ³ 0

  23. Dual Primal

  24. Hill Climbing Exit measureprogress We have a valid solution. (not necessarily optimal) Value of our solution. Take a step that goes up. Make small local changes to your solution toconstruct a slightly better solution. Initially have the “zero Global Max Can't take a step that goes up. Local Max Problems: Running time? Can our Network Flow Algorithm get stuck in a local maximum? If you take small step,could be exponential time.

  25. Hill Climbing Avoiding getting stuck in a local maximum Bad Execution Good Execution • Made better choices of direction • Hard • Back up an retry • Exponential time • Define a bigger step

  26. Network Flow Can our Simplex Algorithm get stuck in local max? No! Need to prove for every linear program for every choice of steps an optimal solution is found! How?

  27. Primal-Dual Hill Climbing Mars settlement has hilly landscapeand many layers of roofs.

  28. Primal-Dual Hill Climbing Primal Problem: • Exponential # of locations to stand. • Find a highest one. Dual problem: • Exponential # of roofs. • Find a lowest one.

  29. Primal-Dual Hill Climbing Prove: • Every roof is above every location to stand.  R  L height(R)  height(L)  height(Rmin)  height(Lmax) Is there a gap?

  30. Primal-Dual Hill Climbing No Gap Prove: • For every location to stand either: • the alg takes a step up or • the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [ L’ height(L’)  height(L) or  R height(R) = height(L)] or But R  L height(R)  height(L)

  31. Primal-Dual Hill Climbing Prove: • For every location to stand either: • the alg takes a step up or • the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [ L’ height(L’)  height(L)or  R height(R) = height(L)] or ? Can't go up from this location and no matching ceiling. Can't happen!

  32. Primal-Dual Hill Climbing No local maximum! Prove: • For every location to stand either: • the alg takes a step up or • the alg gives a reason that explains why not by giving a ceiling of equal height. i.e.  L [ L’ height(L’)  height(L)or  R height(R) = height(L)] or

  33. Primal-Dual Hill Climbing Exit No Gap Claim: Primal and dual have the same optimal value.height(Rmin) = height(Lmax) Proved: R  L, height(R)  height(L) Proved: Alg runs until it provides Lalg and Ralgheight(Ralg) = height(Lalg) height(Rmin)  height(Ralg) = height(Lalg)  height(Lmax) height(Rmin)  height(Lmax) Lalgwitness that height(Lmax)is no smaller. Ralgwitness that height(Lmax)is no bigger.

  34. Duality minimize subject to Cj Xj Mi,j Ni Xj ³ Linear Program Minimize: CTX Subject to: MX³ N ³ Xj 0 • n variable xj that we are looking for values of. • An optimization function • Each has a variable has a coefficient cj. • The dot product CTX gives one value to minimize. • m constraints: • Some linear combination of the variablesmust be at least some set value. • i MiX ³ Ni • Generally implied that variables are positive.

  35. Duality maximize minimize subject to subject to Cj Cj Mi,j Mi,j Ni Ni Xj Xj Xj  ³  Linear Program Minimize: CTX Subject to: MX³ N = Maximize: CTX Subject to: MX³ N Xj ³ = • These are the linear programs in “standard” form • Minimize CTX hence X subject to X ³ • Maximize CTX hence X subject to X  • But you could mix and match.

  36. For every primal linear program, we define its dual linear program. Duality Primal Linear Program Minimize: CTX Subject to: MX³ N minimize subject to Cj Xj Mi,j Ni Xj ³ ? ³ Yi DualLinear Program 0 subject to Yi MTj,i Ni Yi Everything is turned upside down. • The matrix of coefficients is transposed • For each constraint, a variable. • Form the objective function vector from the constraint vector. • Generally, the constraint is ‘’ and then the variable is Yi  0

  37. For every primal linear program, we define its dual linear program. Duality Primal Linear Program Minimize: CTX Subject to: MX³ N minimize subject to Cj Xj Mi,j Ni Xj ³ = ? >=< Yi DualLinear Program 0 subject to Yi MTj,i Ni Yi Everything is turned upside down. • The matrix of coefficients is transposed • For each constraint, a variable. • Form constraint vector from the objective function vector. • Generally, the constraint is ‘’ and then the variable is Yi  0 • But if it is ‘=’, then the variable Yiis unconstrained.

  38. For every primal linear program, we define its dual linear program. Duality Primal Linear Program Minimize: CTX Subject to: MX³ N minimize subject to Cj Xj Mi,j Ni Xj ³ DualLinear Program maximize subject to Yi MTj,i Ni Cj Yi maximize NT.y MT.y  CT  Everything is turned upside down. • The matrix of coefficients is transposed • For each constraint, a variable. • For each variable, a constraint. • Form constraint vector from the objective function vector. • Max  Min and    ?

  39. For every primal linear program, we define its dual linear program. Duality Primal Linear Program Minimize: CTX Subject to: MX³ N minimize subject to Cj Xj Mi,j Ni Xj ³ DualLinear Program maximize subject to Yi MTj,i Ni Cj Yi maximize NT.y MT.y  CT  Everything is turned upside down. • Max Flow  Min Cut • Buyer of nutrients  Seller of nutrients in fruit in vitamins

  40. For every primal linear program, we define its dual linear program. Duality Primal Linear Program Minimize: CTX Subject to: MX³ N minimize subject to Cj Xj Mi,j Ni Xj ³ DualLinear Program maximize subject to Yi MTj,i Ni Cj Yi maximize NT.y MT.y  CT  Every solution X of the primal is above every solution Y of the primal. X YCTX NTY • CXminNYmax • We will prove equality. Dual of the dual is itself!

  41. The Nutrition Problem • Each fruit contains different nutrients • Each fruit has different cost An apple a day keeps the doctor away – but apples are costly! A customer’s goal is to fulfill daily nutrition requirements at lowest cost.

  42. The Nutrition Problem (cont’d) • Let’s take a simpler case of just apples and bananas. • Must take at least 100 units of Calories & 90 units of Vitamins for good nutrition. • A customer’s goal is to buy fruits in such a quantity that it minimizes cost but fulfills nutrition. Cost Function Cost = 5x1 + 7x2 Constraint Functions C1: 2x1 + 4x2³ 100 C2: 3x1 + 3x2³ 90 x1, x2³ 0

  43. Real life problems may have many variables and constraints ! The Nutrition Problem (cont’d) • Matrix Representation Constraints: 2x1 + 4x2³ 100 3x1 + 3x2³ 90 Non-negativity: x1, x2³ 0 Cost function = 5x1 + 7x2 Cj Xj Mi,j Xj Ni ³

  44. Daily nutrition Cost of each fruit Quantity of each fruit Coefficients in each column represent the amount of nutrients in a particular food Primal LP: minimize C.x Q.x ³ N Semantics of Duality A customer’s goal is to buy fruits in such a quantity that it minimizes cost but fulfills nutrition. Cj Xj Mi,j Xj Ni ³

  45. Imagine a salesman trying to sell supplements for each fruit. Dual LP: maximize NT.y QT.y  CT Semantics of Duality Ni Yi Mj,i Yi Cj Cost of each fruit Daily nutrition  But what are Yis in the dual ? Price of each nutrient! Coefficients in each row represent the amount of nutrients in a particular fruit

  46. Semantics of Duality • Primal Problem: A customer’s goal is to buy fruits in such a quantity that it minimizes cost but fulfills nutrition. • Dual Problem:A salesman goal is to set a price on each nutrient, so that it maximizes profit but his supplements are cheaper than fruits. (Otherwise who will buy them?!) Primal (Customer) Dual (Salesman)

  47. Primal-Dual Hill Climbing No Gap Prove: • For every solution x of the primal either: • the alg takes a step up to a better primal • the alg gives a reason that explains why not by giving a solution y of the dual of equal value. i.e.  x [ x’ CTx’ > CTx or  y CTx = NTy] or But x  y Cx  Ny

  48. Primal-Dual Hill Climbing Prove: • For every solution x of the primal either: • the alg takes a step up to a better primal • the alg gives a reason that explains why not by giving a solution y of the dual of equal value. i.e.  x [ x’ CTx’ > CTx or  y CTx = NTy] or ? Can't go up from this location and no matching ceiling. Can't happen!

  49. Primal-Dual Hill Climbing No local maximum! Prove: • For every solution x of the primal either: • the alg takes a step up to a better primal • the alg gives a reason that explains why not by giving a solution y of the dual of equal value. i.e.  x [ x’ CTx’ > CTx or  y CTx = NTy] or

  50. Primal-Dual Hill Climbing Exit No Gap • Claim: Primal and dual have the same optimal value.CTxmin= NTymax xalgwitnesses that CTxminis no smaller. yalgwitnesses that CTxminis no bigger.

More Related