~ Presented by: Michael Gould CS 275

1. ~ Presented by: Michael Gould CS 275 December 7, 2010 Schreiber, Yevgeny.Value-Ordering Heuristics: Search Performance vs. Solution Diversity.In: D. Cohen (Ed.) CP 2010, LNCS 6308, pp. 429-444. Springer- Heidelberg (2010).

2. Introduction • Problem: Given a CSP, generate a large number of diverse solutions as fast as possible. • Trade-off: Solution search performance (time) vs. the diversity of the generated solutions. • Tool: Use value-ordering heuristics to help solve problem.

3. General Solution Search Method • Input: A CSP (X, D, C) • a) Repeatedly select an unassigned variable x. • b) Try to assign to x one of the remaining values in its domain. • c) Propagate every constraint c that involves x by removing the conflicting values in the domains of other unassigned variables. • d) Backtrack if necessary.

4. Value-Ordering Heuristics • A value-ordering heuristic determines which value is selected by the backtracking search algorithm for a given unassigned variable. • Ex. Survivors-first heuristics use simple statistics accumulated during the search to select the value that has been involved in the least number of conflicts.

5. Solution Diversity • Solution Diversity: Requirement that the multiple solutions we find be as “different” from each other as possible. • Solution Diversity is different from solution distribution. Consider a solution space composed of a small subset S1 of solutions where each variable is assigned a different value and a much larger set of solutions S2 so that only a single variable is assigned a different value in each solution.

6. Solution Diversity (continued) • There is a trade-off between the search performance and the solution diversity. • The MAXDIVERSEkSET problem is to compute k maximally diverse solutions of a given CSP. • One measure of the distance between a pair of solutions is Hamming Distance: Given solution s = <s1,...,sn> and solution s' = <s1',...,sn'> we define Hi(s,s') = 1 if si ≠ si' and 0 otherwise for 1≤i≤n. The Hamming distance is the sum of all Hi values.

7. Automatic Test Generation Problem (ATGP) • A real-world example of a CSP where solution diversity is important. • The problem is to generate automatically a valid test for a given hardware specification which is a sequence of a large number of instructions. • These instructions must be diverse in order to trigger as many as possible different hardware events. • Time is important: thousands of tests have to be generated even for a small subset of a modern hardware specification • We consider the overall quality of the whole test-base: size and diversity of tests.

8. The RANDOM Heuristic • Simply select a uniformly random value from the domain of a variable. • Often achieves relatively high solution diversity and performs relatively fast. Reason: in these problems there is a relatively large number of values in every domain whose selection does not lead to a conflict. • Procedure itself of randomly selecting a value is very fast.

9. (1) The LeastFails Heuristic

10. (2) BestSuccessRatio Heuristic

11. (3) & (4) Probabilistic Versions

12. Heuristics (5) and (6)

13. Initial Ordering of Values • No information is available at the beginning of the search for a first solution. Thus, all the values in each domain are initially unordered. • The heuristics can only order values that have already been attempted to be assigned to variables in the past. It must be decided what probability should be used for the values whose order cannot be determined. • We define the conservativenessof a heuristic. The more conservative a heuristic is, the lower the probability to select an unordered value. We can define the conservativeness C(H) of a heuristic H separately for each variable whose domain contains at least one unordered value. • Low C(H) value → initially random behavior but prevents situations where many unordered values are never selected.

14. Heuristic Parameters • α : Controls the conservativeness of a heuristic. Represents an “initial score” of an unordered value u. • β : Used in the probabilistic heuristics to control the aggressiveness of the heuristic (the distribution of probabilities). • γ : A tie-range parameter used to smooth differences between “sufficiently close” values for the heuristics LeastFails and BestSuccessRatio.

15. Experiments • (1) Randomly generated problems. <a,b, c, d>: a = number of variables, b = domain size of each variable, c = number of binary constraints, d = number of incompatible value pairs in each constraint. Tested <50, 10, 225, 35> and <50, 10, 225, 37> with 30 problems in each set. Looked for 30 solutions for each problem, and solved using the variable-ordering that picked the variable with the minimal domain. • (2) ATGP problems. Thousands of variables and constraints that model Intel 64 and IA-32 processor architecture. <n, m>: n = # of problems in set, m = avg. # of instructions required to generate for each problem. Tested <15,21> and <26, 7>.

16. Results • Results summarized in tables. Table entries: <A, B, C, D, E> A: the acronym of the heuristic. B: the value of α used as the “initial score” of an unordered value. C: the value of either β or γ. D: ratio of the avg. time for the heuristic to find a single solution over time required for the random heuristic. E: ratio of avg. Hamming distance between solutions found by the heuristic over distance achieved by random.

20. Analysis • Entries in table sorted by D/E. Lower the ratio → better the performance/quality. Heuristic considered better than RANDOM if D <= E. • Hard to achieve better solution diversity than RANDOM. Only a few entries have E > 1. • Many heuristic configurations run much faster than RANDOM. (e.g. LeastFails can run 10 to 20 times faster). • Very high speedup → significant loss of solution diversity. • Some good heuristic configurations that achieve significant speedup and hardly any loss of solution diversity at top of tables.

21. Analysis (continued) • LeastFails and BestSuccessRatio: Usually achieve very high speed-up, but often accompanied by a significant loss of solution diversity. In randomly generated problems, loss of solution diversity was lower than speed-up gain. • ProbMostFails and ProbWorstSuccessRatio: Usually do not achieve a better solution diversity than Random, but can be much slower. • ProbLeastFails and ProbBestSuccessRation: Often achieve a moderate speed-up without sacrificing much solution diversity.

22. Questions • Do the experimental results hold for larger problems (hundreds of variables, larger domain sizes)? • Do the experimental results hold for other real-world CSP problems? • Do other variable orderings achieve better performance? • Can we combine value-ordering heuristics with inference methods such as search look-ahead or arc-consistency or is the overhead too high? • Is there a consensus on which value-ordering heuristic is the best?