1 / 45

An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques

July 30, 2003. An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques for Solving Finite Constraint Satisfaction Problems. Zheying Jane Yang. Constraint Systems Laboratory Department of Computer Science and Engineering University of Nebraska-Lincoln.

wyman
Download Presentation

An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. July 30, 2003 An Empirical Study of the Performance of Preprocessing and Look-ahead Techniques for Solving Finite Constraint Satisfaction Problems Zheying Jane Yang Constraint Systems Laboratory Department of Computer Science and Engineering University of Nebraska-Lincoln Under the supervision of Prof. B.Y. Choueiry

  2. Outline • Motivation & results • Background • Experimental design and empirical study • Results and analysis • Conclusions & relation to previous work • Summary of contributions • Future work

  3. Search w/ look-ahead CSP Preprocessing Solution(s) SmallerCSP 1. Motivation • CSP used to model NP-complete decision problems • Search (exponential) is necessary, improved with • Preprocessing algorithms: remove inconsistent combinations prior to search • Look-ahead algorithms: remove inconsistencies during search

  4. Algorithms studied & goal • Several competing algorithms • Preprocessing: removes inconsistencies prior to search • Arc-consistency: AC3, AC2001 • Neighborhood inverse consistency (NIC), requires search • Look-ahead: filters search space during search • Forward checking (FC) • Maintaining arc-consistency (MAC) • Controversies about their relative performance exist Our goal is to characterize empirically the relative performance of combinations of preprocessing and look-ahead schemes as a function of the CSP’s constraint probability & tightness

  5. Issue Claims AC2001 is better Bessière & Régin 2001 AC3 vs AC2001 Preprocessing AC vs NIC NIC is better Freuder & Elfe 1996 Haralick & Elliott 1980 FC is better Nadel 1989 Sabin & Freuder 1994 MAC is much better MAC vs FC Look-ahead Bessière & Régin 1996 Grant 1997 Gent & Prosser 2000 No winner performs extremely well on all types of CSPs MAC is the winner in large spase CSPs FC is the winner in dense CSPs Our results:We specify when the above claims hold or not Current beliefs & results

  6. 2. Background

  7. X {a,b} {(b,c), (b,d)} {(b,e), (a,f)} Z Y {c,d} {e,f} {(c,e), (d,f)} What is a CSP? • A CSP problem is defined as a triple P=(V,D,C) • The goal is to find One solution All solutions

  8. Solving CSPs with systematic BT search • Backtrack search is sound and complete, but exponential • expands a partial solution • explores every combination systematically • Improve performance of search • Preprocessing (constraint filtering/propagation) • Look-ahead (constraint filtering during search) • Backtracking: chronological/intelligent • Variable & value ordering: static/dynamic, heuristic • etc.

  9. Solving a CSP Look-ahead Preprocessing • We study the performance of • Preprocessing • Hybrid search = preprocessing x look-ahead algorithms Following an assignment, removes values from the domains of the future variables that are inconsistent with the current assignment. Enforces different levels of local consistency by deleting inconsistent values from variable domains Algorithms: AC3, AC2001, NIC Algorithms: FC & MAC

  10. {B} V1 V2 V3 {R} {G} V1 V1  {R, G,B} {R, G,B}  V2 V3 V2 V3 {R, G} {G} {R, G} {G} V1 {R, B} V2 V3 {R, G} {G} V1 {R, B}  V2  V3 {R} {G} Preprocessing a graph coloring problem

  11. Start V1 V2 V3 V1 {R} V2 V3 {R,G} {G} Solving CSP using forward checking (FC)

  12. Start V1 V2 V3 V1 {R} V2 V3 Domain wiped-out backtrack {G} {G} Solving CSP using forward checking (FC)

  13. Start V1 V2 V3 {G} V1 V2 V3 Domain wiped-out backtrack {R, G} {G} Solving CSP using forward checking (FC)

  14. Start V1 V2 V3 V1 {B} V2 V3 {R, G} {G} Solving CSP using forward checking (FC)

  15. Solving CSP using forward checking (FC) Start V1 V2 V3 V1 {B} V2 V3 {G} {R}

  16. Start V1 V2 V3 Solution! V1 {G} V2 V3 {R} {G} Solving CSP using forward checking (FC)

  17. 3. Experimental design & empirical study

  18. Algorithms tested Preprocessing  Look-ahead AC3 AC2001 FC MAC-AC3 MAC-AC2001 FC MAC-AC3 MAC-AC2001   NIC • Preprocessing: 5 algorithms • Look-ahead: 3 algorithms • Hybrid search: 7 combinations

  19. Working assumptions • Considered only binary constraints • Restricted to finding first solution • Restricted to chronological backtracking • Use least domain (LD) as variable ordering heuristic • Variable ordering done dynamically • No value ordering heuristic, too costly in general

  20. Problems tested • Random CSPs • We need many instances with similar characteristics in order to analyze the performance of search • Real-world problems cannot be controlled by explicit parameters to enable statistical analysis of the performance • Generated instances • connected graphs • instances guaranteed solvable

  21. C Cmax Control parameters < n, d, C, t > • Number of variables n • We choose n = 50 • Domain size d (uniform) • We choose d = 10. Thus, problem size is 1050, relatively large • Constraint tightness t = (uniform) • We vary t = [0.1, 0.2, …, 0.9] and t = [0.05, 0.1, …, 0.95] • Number of constraints C determines constraint probability p= , Cmax= n(n-1)/2 • We vary C = [20, 490] corresponding to p = [0.024, 0.4] • We report C = 30, 50, 80, 120, 130, 245, 340, 490 Disallowedtuples Allpossibletuples

  22. Comparisons • Evaluation criteria • Filtering effectiveness measures reduction of CSP size • Number of constraint checks #CC measures filtering effort • Number of nodes visited #NV measures backtracking effort • CPU time [ms] measures overall performance Since constraints are defined in extension, CPU time reflects #CC • Preprocessing: Filtering effectiveness, #CC, CPU time • Hybrid search: #CC, #NV, CPU time

  23. Sampling and code characteristics • Evaluated performance by running each algorithm • on 30 CSP instances of equal characteristics, • calculating average & standard deviation values for each evaluation criterion • Generated approximately 6,000 CSP instances • Implemented in JAVA: • 21 JAVA classes • 4,000 lines of code • Experiments carried out on PrairieFire.unl.edu

  24. 4. Results and analysis Part I: Preprocessing Part II: Hybrids

  25. Preprocessing: filtering effectiveness NIC reduces search space at a large amount

  26. Preprocessing results » Better than,  comparable

  27. Preprocessing: #CC, sparse CSPs CC (AC3) > CC (AC2001) CC (NICx) >> CC (AC3, AC2001)

  28. Preprocessing: #CC, denser CSPs When p=0.2, NICx are all costly NIC should never be combined with MAC

  29. Preprocessing results » Better than,  comparable

  30. Preprocessing: CPU time When p > 0.2, NIC-x is too costly

  31. Preprocessing results » Better than,  comparable

  32. Preprocessing: summary • AC2001 is not significantly better than AC3, and is not worth the extra data structures In general, we disagree with Bessière and Régin • NIC-x: powerful filtering, but too costly. Use it on large problems when checking constraints is cheap

  33. Hybrids: #CC, p = 5%, 11% When p < 0.05: ACx is not effective,ACx-FC are costly, NIC-FC-FC is OK, NIC-MAC-x is best When p > 0.10: avoid MAC, stick with FC

  34. Hybrids results » Better than,  comparable #CC & CPU time

  35. Hybrids: #CC, p= 15%, 28% As p increases: MAC deteriorates, NIC becomes expensive, use ACx-FC

  36. Hybrids results » Better than,  comparable #CC & CPU time

  37. Hybrids: #NV At phase transition, NIC and MAC • do powerful filtering • but influence of MAC is stronger

  38. Hybrids: summary » Better than,  comparable #CC & CPU time

  39. 5. Conclusions • Preprocessing • AC3 vs. AC2001: AC2001 is not significantly better than AC3, and is not worth the extra data structures In general, we disagree with Bessière and Régin • NIC-x: powerful filtering, but too costly Use it on large problems when checking constraints is cheap • Look-ahead • MAC vs. FC: performance depends on constraint probability and tightness. MAC only wins in low p and high t In general we disagree with results of Sabin, Freuder, Bessière & Régin

  40. Relation to previous work (I) • NIC is better than ACFreuder & Elfe 1996 The instances tested were all with low probability (p < 0.05). In this region, AC is ineffective. • MAC is better than FCSabin & Freuder 1994 They tested CSPs with low probability (p = 0.018-0.09), and relatively high constraint-tightness (t = 0.15 – 0.675). In our study, MAC is effective in this region, but not outside it. • MAC is better than FCBessière & Régin 1996 The instances tested were also in the region of low probability (p = 0.017, 0.024, 0.074, 0.08, 0.1, 0.12, 0.15), except instance#1 and instance#2, with relative high probability (p = 0.3, and 0.84). But here they test only 2 instances

  41. Relation to previous work (II) • Gent & Prosser 2000 questioned validity of previous results on MAC. They concluded that: • in large, sparse CSPs with tight constraints, MAC is winner • in dense CSPs with loose constraints, FC is winner. • Grant 1997 showed that FC is winner on small CSPs with all range of probabilities • All concluded that: “A champion algorithm which perform extremely well on all types of problems does not exist.” • Our characterizations are more thorough and precise.

  42. 6. Summary of contributions • Random generator that guarantees solvability • Empirical evaluation of the performance of 7 combinations of preprocessing and look-ahead • Uncovered (restricted) validity conditions of previously reported results • Summarized best working conditions for preprocessing and look-ahead algorithms • Developed a Java library with 7 hybrid algorithms

  43. 7. Directions for future work • Compare to other, less common filtering algorithms, e.g. SRPC, PC, SAC, Max-RPC Debryune & Bessière 2001 • Combining these preprocessing algorithms with intelligent backtrack search algorithms • Validate results on larger CSPs, real-world applications, non-binary • Test and analyze the effect of the topology of constraint networks on the performance of search

  44. Acknowledgments Dr. Berthe Y. Choueiry (advisor) Dr. Sebastian Elbaum Dr. Peter Revesz Ms. Catherine L. Anderson Mr. Daniel Buettner Ms. Deborah Derrick (proof reading) Mr. Eric Moss Mr. Lin Xu Ms. Yaling Zheng Mr. Hui Zou

  45. THANK YOU!

More Related