1 / 41

Probabilistic Modeling for Combinatorial Optimization

Probabilistic Modeling for Combinatorial Optimization. Scott Davies School of Computer Science Carnegie Mellon University Joint work with Shumeet Baluja. Combinatorial Optimization. Maximize “evaluation function” f(x) input: fixed-length bitstring x output: real value

Download Presentation

Probabilistic Modeling for Combinatorial Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Probabilistic Modeling for Combinatorial Optimization Scott Davies School of Computer Science Carnegie Mellon University Joint work with Shumeet Baluja

  2. Combinatorial Optimization • Maximize “evaluation function” f(x) • input: fixed-length bitstring x • output: real value • x might represent: • job shop schedules • TSP tours • discretized numeric values • etc. • Our focus: “Black Box” optimization • No domain-dependent heuristics x=1001001... f(x) 37.4

  3. Most Commonly Used Approaches • Hill-climbing, simulated annealing • Generate candidate solutions “neighboring” single current working solution (e.g. differing by one bit) • Typically make no attempt to model how particular bits affect solution quality • Genetic algorithms • Attempt to implicitly capture dependency of solution quality on bit values by maintaining a population of candidate solutions • Use crossover and mutation operators on population members to generate new candidate solutions

  4. Using Explicit Probabilistic Models • Maintain an explicit probability distribution P from which we generate new candidate solutions • Initialize P to uniform distribution • Until termination criteria met: • Stochastically generate K candidate solutions from P • Evaluate them • Update P to make it more likely to generate solutions “similar” to the “good” solutions • Several different choices for what sorts of P to use and how to update it after candidate solution evaluation

  5. Probability Distributions Over Bitstrings • Let x = (x1, x2, …, xn), where xi can take one of the values {0, 1} and n is the length of the bitstring. • Can factorize any distribution P(x1…xn) bit by bit: P(x1,…,xn) = P(x1) P(x2 | x1) P(x3 | x1, x2)…P(xn|x1, …, xn-1) • In general, the above formula is just another way of representing a big lookup table with one entry for each of the 2n possible bitstrings. • Obviously too many parameters to estimate from limited data!

  6. Representing Independencies withBayesian Networks • Graphical representation of probability distributions • Each variable is a vertex • Each variable’s probability distribution is conditioned only on its parents in the directed acyclic graph (“dag”) Wean on Fire P(F,D,I,A,H) = P(F) * P(D) * P(I|F) * P(A|F) * P(H|D,I,A) F Ice Cream Truck Nearby Fire Alarm Activated D I A Office Door Open H Hear Bells

  7. x1 x2 x3 xn x1 x2 x3 xn “Bayesian Networks” for Bitstring Optimization? P(x1,…,xn) = P(x1) P(x2 | x1) P(x3 | x1, x2)…P(xn|x1, …, xn-1) Yuck. Let’s just assume all the bits are independent instead. (For now.) P(x1,…,xn) = P(x1) P(x2) P(x3)…P(xn) Ahhh. Much better.

  8. Population-Based Incremental Learning • Population-Based Incremental Learning (PBIL) [Baluja, 1995] • Maintains a vector of probabilities: one independent probability P(xi) for each bit xi. • Until termination criteria met: • Generate a population of K bitstrings from P • Evaluate them • Use the best M of the K to update P as follows: if xi is set to 1 or if xi is set to 0 • Optionally, also update P similarly with the bitwise complement of the worst of the K bitstrings • Return best bitstring ever evaluated

  9. PBIL vs. Discrete Learning Automata • Equivalent to a team of Discrete Learning Automata, one automata per bit. [Thathachar & Sastry, 1987] • Learning automata choose actions independently, but receive common reinforcement signal dependent on all their actions • PBIL update rule equivalent to linear reward-inaction algorithm [Hilgard & Bower, 1975] with “success” defined as “best in the bunch” • However, Discrete Learning Automata typically used previously in problems with few variables but noisy evaluation functions

  10. PBIL vs. Genetic Algorithms • PBIL originated as tool for understanding GA behavior • Similar to Bit-Based Simulated Crossover (BSC) [Syswerda, 1993] • Regenerates P from scratch after every generation • All K used to update P, weighted according to probabilities that a GA would have selected them for reproduction • Why might normal GAs be better? • Implicitly capture inter-bit dependencies with population • However, because model is only implicit, crossover must be randomized. Also, limited population size often leads to premature convergence based on noise in samples.

  11. Four Peaks Problem • Problem used in [Baluja & Caruana, 1995] to test how well GAs maintain multiple solutions before converging. • Given input vector X with N bits, and difficulty parameter T: • FourPeaks(T,X)=MAX(head(1,X), tail(0,X))+Bonus(T,X) • head(b,X) = # of contiguous leading bits in X set to b • tail(b,X) = # of contiguous trailing bits in X set to b • Bonus(T,X) = 100 if (head(1,X)>T) AND (tail(0,X) > T), or 0 otherwise

  12. Four Peaks Problem Results • 80 GA settings tried; best five shown here • Average over 50 runs

  13. Large-Scale PBIL Empirical Comparison • See table in other slide

  14. Modeling Inter-Bit Dependencies • How about automatically learning probability distributions in which at least some dependencies between variables are modeled? • Problem statement: given a dataset D and a set of allowable Bayesian networks {Bi}, find the Bi with the maximum posterior probability:

  15. Equations. (Mwuh hah hah hah!) • Where: • d j is the jth datapoint • dij is the value assigned to xi by d j. • Pi is the set of xi’s parents in B • is the set of values assigned to Pi by d j • P’ is the empirical probability distribution exhibited by D

  16. Mmmm...Entropy. • Fact: given that B has a network structure S, the optimal probabilities to use in B are just the probabilities in D, i.e. P’. So: Pick B to Maximize: Pick S to Maximize:

  17. x17 x32 x20 Entropy Calculation Example x20’s contribution to score: 80 total datapoints where H(p,q) = p log p + q log q.

  18. x17 x22 x5 x2 x13 x19 x6 x14 x8 Single-Parent Tree-Shaped Networks • Now let’s allow each bit to be conditioned on at most one other bit. • Adding an arc from xj to xi increases network score by H(xi) - H(xi|xj) (xj’s “information gain” with xi) = H(xj) - H(xj|xi) (not necessarily obvious, but true) = I(xi, xj)Mutual information between xi and xj

  19. Optimal Single-Parent Tree-Shaped Networks • To find optimal single-parent tree-shaped network, just find maximum spanning tree using I(xi, xj) as the weight for the edge between xi and xj. [Chow and Liu, 1968] • Start with an arbitrary root node xr. • Until all n nodes have been added to the tree: • Of all pairs of nodes xin and xout, where xin has already been added to the tree but xout has not, find the pair with the largest I(xin, xout). • Add xout to the tree with xin as its parent. • Can be done in O(n2) time (assuming D has already been reduced to sufficient statistics)

  20. Optimal Dependency Trees for Combinatorial Optimization [Baluja & Davies, 1997] • Start with a dataset D initialized from the uniform distribution • Until termination criteria met: • Build optimal dependency tree T with which to model D. • Generate K bitstrings from probability distribution represented by T. Evaluate them. • Add best M bitstrings to D after decaying the weight of all datapoints already in D by a factor a between 0 and 1. • Return best bitstring ever evaluated. • Running time: O(K*n + n2) per iteration

  21. Tree-based Optimization vs. MIMIC • Tree-based optimization algorithm inspired by Mutual Information Maximization for Input Clustering (MIMIC) [De Bonet, et al., 1997] • Learned chain-shaped networks rather than tree-shaped networks • Dataset: best N% of all bitstrings ever evaluated • +: dataset has simple, well-defined interpretation • -: have to remember bitstrings • -: seems to converge too quickly on some larger problems x4 x17 x8 x5

  22. MIMIC dataset vs. Exp. Decay dataset • Solving a system of linear equations the hard way • Error of best solution as a function of generation #, averaged over 10 problems • 81 bits

  23. Graph-Coloring Example • Noisy Graph-Coloring example • For each edge connected to vertices of different colors, add +1 to evaluation function with probability 0.5

  24. Peaks problems results • Oops. Forgot this slide.

  25. Checkerboard problem results • 16*16 grid of bits • For each bit in middle 14*14 subgrid, add +1 to evaluation for each of four neighbors set to opposite value Genetic Algorithm (avg: 740) PBIL (avg: 742) Chain (avg: 760) Tree (avg: 776)

  26. Linear Equations Results • 81 bits again; 9 problems, each with results averaged over 25 runs • Minimize error

  27. Modeling Higher-Order Dependencies • The maximum spanning tree algorithm gives us the optimal Bayesian Network in which each node has at most one parent. • What about finding the best network in which each node has at most K parents for K>1? • NP-complete problem! [Chickering, et al., 1995] • However, can use search heuristics to look for “good” network structures (e.g., [Heckerman, et al., 1995]), e.g. hillclimbing.

  28. - l|B| Scoring Function for Arbitrary Networks • Rather than restricting K directly, add penalty term to scoring function to limit total network size • Equivalent to priors favoring simpler network structures • Alternatively, lends itself nicely to MDL interpretation • Size of penalty controls exploration/exploitation tradeoff l: penalty factor |B|: number of parameters in B

  29. Bayesian Network-Based Combinatorial Optimization • Initialize D with C bitstrings from uniform distribution, and Bayesian network B to empty network containing no edges • Until termination criteria met: • Perform steepest-ascent hillclimbing from B to find locally optimal network B’. Repeat until no changes increase score: • Evaluate how each possible edge addition, removal or deletion would affect penalized log-likelihood score • Perform change that maximizes increase in score. • Set B to B’. • Generate and evaluate K bitstrings from B. • Decay weight of datapoints in D by a. • Add best M of the K recently generated datapoints to D. • Return best bit string ever evaluated.

  30. Cutting Computational Costs • Can cache contingency tables for all possible one-arc changes to network structure • Only have to recompute scores associated with at most two nodes after arc added, removed, or reversed. • Prevents having to slog through the entire dataset recomputing score changes for every possible arc change when dataset changes. • However, kiss memory goodbye • Only a few network structure changes required after each iteration since dataset hasn’t changed much (one or two structural changes on average)

  31. Evolution of Network Complexity • 100 bits; +1 added to evaluation function for every bit set to the parity of the previous K bits

  32. Summation Cancellation • Minimize magnitudes of cumulative sum of discretized numeric parameters (s1, …, sn) represented with standard binary encoding: Average value over 50 runs of best solution found in 2000 generations

  33. Bayesian Networks: Empirical Results Summary • Does better than Tree-based optimization algorithm on some toy problems • Significantly better on “Summation Cancellation” problem • 10% reduction in error on System of Linear Equation problems • Roughly the same as Tree-based algorithm on others, e.g. small Knapsack problems • Significantly more computation despite efficiency hacks, however. • Why not much better performance? • Too much emphasis on exploitation rather than exploration? • Steepest-ascent hillclimbing over network structures not good enough?

  34. Using Probabilistic Models for Intelligent Restarts • Tree-based algorithm’s O(n2) execution time per generation very expensive for large problems • Even more so for more complicated Bayesian networks • One possible approach: use probabilistic models to select good starting points for faster optimization algorithms, e.g. hillclimbing

  35. COMIT • Combining Optimizers with Mutual Information Trees [Baluja & Davies, 1997b]: • Initialize dataset D with bitstrings drawn from uniform distribution • Until termination criteria met: • Build optimal dependency tree T with which to model D. • Generate K bitstrings from the distribution represented by T. Evaluate them. • Execute a hillclimbing run starting from single best bitstring of these K. • Replace up to M bitstrings in D with the best bitstrings found during the hillclimbing run. • Return best bitstring ever evaluated

  36. COMIT, cont’d • Empirical tests performed with stochastic hillclimbing algorithm that allows at most PATIENCE moves to points of equal value before restarting • Compare COMIT vs.: • Hillclimbing with restarts from bitstrings chosen randomly from uniform distribution • Hillclimbing with restarts from best bitstring out of K chosen randomly from uniform distribution • Genetic algorithms?

  37. COMIT: Example of Behavior • TSP domain: 100 cities, 700 bits Tour Length * 103 Evaluation Number * 103

  38. COMIT: Empirical Comparisons • Each number is average over at least 25 runs • Highlighted: better than each non-COMIT hillclimber with P > 95% • AHCxxx: pick best of xxx randomly generated starting points • COMITxxx: pick best of xxx starting points generated by tree

  39. Summary • PBIL uses very simple probability distribution from which new solutions are generated, yet works surprisingly well • Algorithm using tree-based distributions seems to work even better, though at significantly more computational expense • More sophisticated networks: past the point of diminishing marginal returns? “Future research” • COMIT makes tree-based algorithm applicable to much larger problems

  40. Future Work • Making algorithm based on complex Bayesian Networks more practical • Combine w/simpler search algorithms, ala COMIT? • Applying COMIT to more interesting problems • WALKSAT? • Using COMIT to combine results of multiple search algorithms • Optimization in real-valued state spaces. What sorts of PDF representations might be useful? • Gaussians? • Kernel-based representations? • Hierarchical representations? • Hands off :-)

  41. Acknowledgements • Shumeet Baluja • Doug Baker, Justin Boyan, Lonnie Chrisman, Greg Cooper, Geoff Gordon, Andrew Moore, …?

More Related