Loading in 2 Seconds...

Create Presentation
Download Presentation

Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming

Loading in 2 Seconds...

Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming

0 Views

Download Presentation
Download Presentation
## Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Artificial Intelligence: Representation and Problem**SolvingOptimization (2): Linear Programming 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu.edu Wean Hall 4126**Recap**• Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum s.t. Fei Fang**Recap**• Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum • Nonconvex but continuous optimization • Gradient descent leads to local optima s.t. Reference: Wikipedia; Berkeley CE191**Recap**• Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum • Nonconvex but continuous optimization • Gradient descent leads to local optima • This section: is a linear function and is defined by a set of linear constraints s.t. Reference: Wikipedia; Berkeley CE191**Outline**• Linear Program • Simplex Algorithm • LP Duality Fei Fang**Linear Program: Definition**• Linear Program: • A special case of convex optimization problem • An optimization problem whose optimization objective is a linear function and feasible region is defined by a set of linear constraints s.t. Note: can also be minimization Fei Fang**Linear Program: Example**• Example: Maximize Profit in Manufacturing s.t. s.t. Fei Fang**Linear Program: Example**• Example: Maximize Profit in Manufacturing s.t. Fei Fang**Quiz 1: Linear Program**• Which constraints determine point B? Fei Fang**Linear Program: Definition**• Feasible region defined by • define a halfspace • Feasible region is the intersection of halfspaces, i.e., a linear polytope s.t. Fei Fang**Linear Program: Definition**• Objective function • Represent a direction of optimization • Generally, optimum always occur on a corner, i.e., a vertex of the feasible region s.t. Fei Fang**Linear Program: Definition**• Infeasible LP • Feasible region is empty • Unbounded LP • Unbounded polytope • Optimal value s.t. s.t. Fei Fang**Linear Program: How to Solve**• At least one vertex of the polytope is an optimal solution • If is a convex combination of two different feasible points , then the objective value at either or is no worse than the value at Fei Fang**Linear Program: How to Solve**• Naïve approach • Find all vertices • Pick the one with the best objective value How? Fei Fang**Linear Program: How to Solve**• Find vertex • linearly independent linear equations uniquely determines a point in • Choose inequality constraints and set as equality • Solve the linear system s.t. Fei Fang**Linear Program: How to Solve**• Find vertex for LP in standard form • Any LP can be converted into standard form • If maximization, convert to minimization • For any constraint , add slack variables and get s.t. s.t. Fei Fang**Linear Program: How to Solve**• Find vertex for LP in standard form Where • Focus on • If rows of are not linearly independent, apply basic row operation to remove redundant constraints. There is no solution if it leads to contradicting constraints. • If : Unique point . No optimization needed. • If : Infinite points satisfying • So we only consider and s.t. Fei Fang**Linear Program: How to Solve**• Find vertex of the feasible region ,, equality constraints, inequality constraints • Choose a set of variables, denoted as • Solve the following linear system, get a solution • is a vertex of the feasible region if satisfies remaining constraints , how many choices? Only variables in can take non-zero values denotes subselecting columns of based on , denote subselecting element of Fei Fang**Linear Program: How to Solve**Only variables in can take non-zero values • Choose • Solve linear system s.t. Fei Fang**Linear Program: How to Solve**Only variables in can take non-zero values • Choose • Solve linear system What if add a constraint ? s.t. Fei Fang**Linear Program: How to Solve**s.t. feasible Corresponds to vertex B in x-y space feasible Corresponds to vertex D in x-y space infeasible Fei Fang**Linear Program: How to Solve**s.t. infeasible feasible Corresponds to vertex A in x-y space feasible Corresponds to vertex (0,0) in x-y space Fei Fang**Linear Program: How to Solve**• Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement s.t. Fei Fang**Linear Program: How to Solve**• Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement • Recall local search and gradient descent • May enumerate almost all the vertices in the worst case, but very efficient in most cases Fei Fang**Simplex Algorithm**• Again, focus on standard form • A vertex is the solution of the linear system for a certain s.t. Fei Fang**Simplex Algorithm**• A single step in simplex algorithm: Given a vertex and corresponding , move to the neighboring vertex with a decrease in objective value Current solution: A neighboring vertex of only has one different element in Fei Fang**Simplex Algorithm**• Consider adjusting toby setting for some while ensuring , and • All has to change accordingly • Denote , then Fei Fang**Simplex Algorithm**• The new objective value is • Rewritten as where is called reduced cost • If , then and the adjustment is worthwhile. The larger the , the more improvement per unit of change • To ensure , we have • Let . So . • When , we have and a neighboring vertex is reached Fei Fang**Simplex Algorithm**Note: In practice, when checking if and (and other tie breaking conditions), we can compare to to avoid numerical issues Fei Fang**Simplex Algorithm**• Example , s.t. • Introduce to • Reduced cost . If increases by , obj value decreases by • . So if increases by , has to decrease by , has to decrease by • can increase by at most , at which point becomes • So the new is Corresponds to vertex D in x-y space Fei Fang**Simplex Algorithm**, • Example s.t. • Introduce to • Reduced cost • . So if increases by , has to decrease by , has to decrease by • can increase by at most , at which point becomes • So the new is Corresponds to vertex B in x-y space • At this point, none of the variables has a negative reduced cost , so optimal solution is found Fei Fang**Simplex Algorithm**• Handling degeneracy: avoid cycling over the indices • Bland’s rule • At each step, choose smallest 𝑗 such that < 0 • For variables that could exit set choose the smallest Fei Fang**Quiz 2**• Consider the following two LPs (LP-L and LP-R) where • Applying simplex algorithm to LP-L with . Denote the optimal solution as . If , then which of the following claims are true about ? • A: is not in the feasible region of LP-R • B: is in the feasible region of LP-R • C: is a vertex of the feasible region of LP-R • D: is an optimal solution of LP-R LP-R LP-L s.t. s.t. Fei Fang**Two-Phase Simplex Algorithm**s.t. • Phase I (Find one vertex): Apply simplex alg to the following LP with initial vertex , get • If , the original LP is infeasible • Phase II (Find best vertex): Apply simplex alg to the original LP with initial vertex s.t. For , flip the sign of the constraints , provides a vertex to begin with Fei Fang**Simplex Algorithm**• Theorem: the simplex algorithm is guaranteed to find a globally optimal solution to a linear program Proof: If simplex returns, then it has found a point where we cannot improve the objective locally; since linear programs are convex, this is a global optimum Because the objective of the simplex improves at each iteration, and since there are a finite (but exponential) number of vertices in the polytope, the algorithm must return after a finite number of steps Fei Fang**Linear Program: How to Solve**• Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement • Recall local search and gradient descent • May enumerate almost all the vertices in the worst case, but very efficient in most cases • (Not covered) Ellipsoid algorithm • Polynomial-time in theory, poor performance in practice Fei Fang**Linear Program: How**• Solve LPs in practice: • All the solvers/algorithms for Convex Optimization problems can be applied • Additional solvers/algorithms for LPs • linprog (MATLAB), linprog (in Python package SciPy) • PuLP (Python) • Cplex, Gurobi • https://www.informs.org/ORMS-Today/Public-Articles/June-Volume-38-Number-3/Software-Survey-Linear-Programming Fei Fang**Linear Program: Note**• Linear program is a continuous optimization problem, but can be viewed as a combinatorial optimization problem: find a corner of the polytope defined by that maximizes the objective function Fei Fang**Linear Program Duality**• Dual problem of an LP: also a linear program • Each dual variable corresponds to a constraint in primal LP • Strong duality theorem: LP and its dual have the same optimal objective value (if feasible and bounded) Primal LP in Standard Form variables, constraints Dual LP variables, constraints s.t. s.t. , Fei Fang**Linear Program Duality**Primal LP in Standard Form Dual LP s.t. s.t. s.t. s.t. Fei Fang**Linear Program Duality**s.t. s.t. Optimal obj Fei Fang**Write the Dual of an LP (Non-Standard form)**s.t. s.t. Fei Fang**Other properties of LP Duality (not required)**• Complementary slackness • Primal constraint is not tight dual variable= • Primal variable is not zero dual constraint is tight • Dual constraint is not tight primal variable= • Dual variable is not zero primal constraint is tight • Optimality Conditions: If are feasible solutions to the primal and dual problems, respectively, then they are optimal solutions to these problems if, and only if, the complementary-slackness conditions hold for both the primal and the dual problems Fei Fang**Summary**Optimization Problems Convex Programs Gradient Descent Linear Programs Simplex Fei Fang**Linear Program: Additional Resources**• Textbook • Applied Mathematical Programming, Chapters 2-4 • By Bradley, Hax, and Magnanti (Addison-Wesley, 1977) • http://web.mit.edu/15.053/www/AMP.htm • Online course • https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/index.htm Fei Fang**Acknowledgment**• Some slides are borrowed from previous slides made by J. Zico Kolter Fei Fang