What good is a scheduling competition insights from the ipc
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

What Good is a Scheduling Competition? - Insights from the IPC PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

What Good is a Scheduling Competition? - Insights from the IPC. Terry Zimmerman Carnegie Mellon University, Robotics Institute 5000 Forbes Avenue, Pittsburgh, PA [email protected] The International Planning Competition: An exemplar for a scheduling competition?.

Download Presentation

What Good is a Scheduling Competition? - Insights from the IPC

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


What good is a scheduling competition insights from the ipc

What Good is a Scheduling Competition? - Insights from the IPC

Terry Zimmerman

Carnegie Mellon University, Robotics Institute

5000 Forbes Avenue, Pittsburgh, PA

[email protected]


The international planning competition an exemplar for a scheduling competition

The International Planning Competition: An exemplar for a scheduling competition?

  • Close (and growing closer) relationship between planning & scheduling

    • How would a scheduling competition distinguish itself? (a variety of ‘scheduling domains’ have been featured in IPC events)

    • Same question relative to the various other competitions (e.g. Benedetti, Pecora, Policella 2007)

  • 5 competitions held to date (1998, 2000, 2002, 2004, 2006)

    • Young enough: initial competition setup/design issues in recent memory

    • Old enough: Good examples of what worked / didn’t work

    • Learning curve over initial years is of interest

  • It’s the only computational competition I have experience with…


Stated general goals of the ipc

Stated general goals of the IPC

  • analyzing and advancing the planning state-of-the-art

  • providing new benchmarks and a representation formalism to aid planner comparison and evaluation

  • emphasizing new research issues and directions

  • promoting applicability of planning technology

  • (disseminating as much performance data as possible to the community)


Overview of the international planning competitions first ipc 1998 pittsburgh pa

Overview of the International Planning CompetitionsFirst IPC: 1998 Pittsburgh, PA

Major Focus:

  • Non-temporal ‘classical’ planning only

  • No explicit resource modeling or metric values

    Domain Language extensions & Domains PDDL introduced, 6 domains

    Competitors: 5 planners entered

Blackbox, STAN, HSP, IPP, SGP

-all but HSP are Graphplan based

Results: No clear-cut winner. ‘Big’ plans: 30-40 steps, Max solution sizes >100 steps


Overview of the international planning competitions second ipc 2000 breckenridge co

Overview of the International Planning CompetitionsSecond IPC: 2000 Breckenridge, CO

Focus:

  • largely ‘classical’ planning, limited metric values

  • 2 tracks: 1) Fully automated 2) Hand tailored

    Minor refinement of PDDL, 5 domains

    Competitors: 17 planners entered

Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS

Results:

Fully automated> Top performers vary by domains –

FF, STAN, MIPs, HSP2, GRT scale over the 5 domains

Hand-tailored> TALplanner dominates (scaled to 500

blocks, ~1.5s), SHOP often gets shorter length plans

Blackbox, FF, STAN, AltAlt, MIPs, HSP2, IPP, PropPlan, GRT, TokenPlan, SHOP, TALplanner, PbR, SystemR, BDDPlan, CHIPS


What good is a scheduling competition insights from the ipc

Overview of the International Planning CompetitionsThird IPC: 2002 Toulouse, Fr.

Focus:

  • extension to temporal planning

  • extension to numeric constraints & fluents

  • 2 tracks: 1) Fully automated 2) Hand tailored

    Extended PDDL to support temporal & numeric features

    Competitors: 14 planners entered


What good is a scheduling competition insights from the ipc

Overview of the International Planning CompetitionsFourth IPC: 2004 Whistler, B.C

Focus:

  • development of benchmark domains close to applications and diverse in structure

  • optimal planners separated from sub-optimal

  • Introduced uncertainty (probabilistic action effects) Limitation: fully observable domains, discrete distr.

  • 2 tracks: 1) Deterministic 2) Probabilistic

PDDL extended for both tracks:

Deterministic –Derived predicates, Limited exogenous events

Probabilistic –Created PPDDL: effects of actions may have discrete

outcome probs & probabilistic initial state literals

Domains: 7 for deterministic track (2 replays from IPC-3), 8 for prob. track

Competitors: 19 deterministic planners:

Optimal –BFHSP, CPT, HSP*-a,Optiplan, SemSyn, SATPLAN-04, TP4-04

Sub-optimal - CRIKEY, FAP, Fast Downward, Fast Diag. Downward, LPG-TD, Macro-FF,

Marvin, Optop, P-MEP, Roadmapper, SGPlan, Tilsapa, YAHSP,

FF, MIPS, & LPG from IPC-3 also run where capable.

10 probabilistic planners: mGPT, Purdue-Humans, Classy, FF-rePlan,

NMRDPP, ProbaPOP, FCPlanner, CERT


What good is a scheduling competition insights from the ipc

Overview of the International Planning CompetitionsFifth IPC: 2006 English Lakes, U.K

Focus: 2 major tracks-

Deterministic: fully deterministic & observable (previously called "classical" planning).  Subtracks- Optimal Satisficing (sub-optimal)

Non-deterministic: 2 subtracks: 1) Conformant planning: nondeterministic problems for which planners must produce a contingency-safe and linear solution. 2) Probabilistic planning: Focus on real-time decision making not complete policies.

PDDL extended for both tracks:

Deterministic –Derived predicates, Limited exogenous events

Probabilistic –Created PPDDL: effects of actions may have discrete

outcome probs & probabilistic initial state literals

Domains: 7 for deterministic track (2 replays from IPC-3),

8 for prob. track


Ipc goals analyzing and advancing the planning state of the art

IPC Goals“analyzing and advancing the planning state-of-the-art”

Visibility for the competition’s focal problems/ algorithms/tracks will likely increase across the diverse & broad scheduling community

  • Relative maturity of planning / scheduling

    • Planning advances have been driven more by perceived need to expand model expressiveness

    • Scheduling advances have come more from imminent and immediate applications


What good is a scheduling competition insights from the ipc

IPC Goals“providing new benchmarks and a representation formalism to aid planner comparison and evaluation”

There are many existing scheduling benchmarks –But tend to be ‘classical’ in that they don’t include breadth of constraints found in practical apps.

There is no existing broadly recognized scheduling domain description language (i.e. no ‘SDDL’)

  • Like PDDL creation of an SDDL may facilitate comparisons of scheduling paradigms across diverse problems

  • Differences: modeling of resources as 1st class objects

Shed light on relative strengths of planning/scheduling approaches to similar problems: Translate the several IPC ‘scheduling’ domains into SDDL….


Ipc goals emphasizing new research issues and directions

IPC Goals“emphasizing new research issues and directions”

  • tasks with uncertain durations and/or outcomes

  • scheduling/ rescheduling to keep pace with execution

  • distributed or multi-agent scheduling

  • trade-offs in schedule robustness vs. quality/utility


Ipc goals promoting applicability of planning technology

IPC Goals“promoting applicability of planning technology”

(arguably) Demonstrating and promoting applicability is a larger concern at this time for planning than scheduling


Ipc goals disseminating as much performance data as possible to the community

IPC Goals“disseminating as much performance data as possible to the community”

IPC experience: few competing systems are effective, let alone dominate, across many domains and tracks

--Even more likely to be the case for scheduling systems, at least in early competitions.

Must motivate competition to generate this data

Performance visibility improves with a successful competition


The tally

The tally:

So expect recruitment for Scheduling Competition committees to get underway shortly (!)


  • Login