1 / 11

International Planning Competition Series

International Planning Competition Series. Derek Long University of Strathclyde, Glasgow, UK. Competition Format. Biennial event organised by different chairs Multiple tracks to showcase different technologies Beyond STRIPS: Temporal/metric/uncertainty/learning

talon
Download Presentation

International Planning Competition Series

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. International Planning Competition Series Derek Long University of Strathclyde, Glasgow, UK

  2. Competition Format • Biennial event organised by different chairs • Multiple tracks to showcase different technologies • Beyond STRIPS: Temporal/metric/uncertainty/learning • Benchmark problems specified by organisers and delivered to competitors “unseen” • Evaluation according to “pre-specified” criteria, selected by organisers

  3. The Good PDDL inside • Common format inputs promoting development and interchange of benchmarks • Excitement and fun factor (so far...) • Multiple organisers setting different agendas • Growth in strength of empirical evaluation of planners • Availability of more systems with improved robustness and common input formats

  4. The Bad • Finding a continuing stream of organisers.... • Rewards shrinking while demands increasing • Splintering of the competition into many small tracks • Maintaining relevance • Community division into the “competition folk” and the rest • Competition becoming an end in itself?

  5. The Ugly • Self-regulation of competitors • Relaxed timing of distribution of competition materials versus retrieval of results • Poorly specified evaluation criteria • Soundbite reportage: false perceptions and wrong lessons

  6. “Research” versus “Applications” • Competitions are not applications • They form a bridge: • Accessible, well specified problems • Closer to research interests • Repeatable environment, open infrastructure • Do competitions have a more important role when applications are still hard to find?

  7. Interest and Access • Entry bar increases • Perceived relevance falls • applications gap begins to fill? Local optimisation: tweaking what works

  8. Motivations • Pushing research by setting challenges • Encourage jumps out of local optima • Taboo search? • Careful winning criteria • Special prizes? • Pushing engineering quality by rewarding robustness • Rewards for good software tools and encouragement/requirement to open source • Education and training: • encourage student participation and lower entry bar • Showcasing • entertainment value, broad appeal, relevance

  9. Lessons for Competitions • Stay fresh • Catch the mood of the moment • Build up resources for future organisers • Identify long-term organisational goals • Be aware of the dangers of competitions becoming • “an application” • “an institution”

  10. What works? PDDL inside • Common format inputs promoting development and interchange of benchmarks • Excitement and fun factor (so far...) • Multiple organisers setting different agendas Drew McDermott Fahiem Bacchus Maria Fox and Derek Long HakanYounes, Michael Littman Stefan Edelkamp and Jeorg Hoffmann Alfonso Gerevini, Alessandro Saetti, PatrikHaslum BlaiBonet, Robert Givan MalteHelmert, Minh Binh Do, IoannisRefanides Dan Bryce, Olivier Buffet, Alan Fern, RoniKhardon

More Related