AIPS 2000 will feature a planning competition designed to compare and evaluate the current state of the art in planning systems. Benchmark your own approach by entering your system. Or participate by submitting an interesting problem domain.
Competition Tracks
The AIPS 2000 planning competition will consist of 3 separate tracks:
Fully automated planners
This track will be very similar to the STRIPS and ADL tracks run at AIPS-98. In particular, domains and problems will be specified using an updated version of the PDDL description language. There will be a range of domains, using a range of the expressive features of PDDL. In particular, a number of the domains will be restricted to actions specified using the STRIPS formalism, while other domains will also involve features like conditional effects and complex preconditions, like the ADL formalism. A planning system need not be able to handle all domains in order enter the competition.
They will be expected to take as input a domain description and a sequence of planning problems. They will have to output correct plans (in a specific format) for as many of these problems as they can solve. The CPU time taken to solve each problem will be recorded, and the final plan will be evaluated under some quality metric (know to the competitors prior to testing). Planning systems will be ranked using various criteria which will include (1) the number of problems solved in each domain, (2) the time required to solve these problems, and (3) the "quality" of the plan produced.
The planning systems can employ various forms of domain preprocessing and analysis. However, all of this processing must be fully automatic. The system must take only the PDDL domain description as input. After the domain description is processed the systems must then proceed to solve the suite of test problems, again fully automatically.
Hand tailored planning systems
This is a new track for the competition. The planning domains will be specified in advance using the PDDL description language. After the domain is released the contestants will have a fixed period of time in which to configure their system for this domain (e.g., if their system takes additional domain dependent information the contestants will have this time to create and add the relevant information). At the end of this period testing will commence. The planning systems must be able to take as input a suite of problems (specified in the PDDL language) and output correct plans (also in a specific format). The planning systems will not, however, be required to take the PDDL domain description as input. Rather during the configuration period the competitors will have to create a domain description suitable for their system. The planning systems will be ranked using some of the same criteria as in the fully automated strips track. It should also be noted that the ranking criteria will attempt to take into account the customization effort required by the competing systems.
Planning with resources
This will be a test track for the competition. That is, it will not be a fully fledged competition track, but rather will be a track that we hope to evolve into a proper competitive track in the future. In this track the planning systems must be able to deal with resources which might include real valued costs and time. We hope to extend PDDL so that we can specify domains involving resources, but some of the test domain may be specified in other formats. The domains will involve actions that consume and produce metric resources. They may also have actions that can be run concurrently and that produce different effects over time. During the testing each problem will have a well specified cost function. The systems must construct correct plans that attempt to optimize this cost function.
Problem Domains
The success of the competition will depend critically on the suite of test domains. We encourage the submission of possible test domains. These domains should be formally specified (preferably in PDDL). These domains will be credited to the submitters and will be archived by ETAI (Electronic Transactions on Artificial Intelligence) as part of their planning and scheduling area.
Tentative Timetable
Jan 7th Competitors must register with the competition chair.
Mar 1st Preliminary rounds of the competition commence.
April 14-17th AIPS-2000 conference. The final rounds of the competition will be run concurrently with the conference.
Further Information
For further information of suggestions please contact the competition coordinator Fahiem Bacchus fbacchus@cs.toronto.edu or visit the planning competition homepage http://www.cs.toronto.edu/aips2000.
In particular, if you interested in entering the competition but are concerned about how your system can fit, please contact the competition chair. We want to allow the competition to encompass as many different approaches to planning as is feasible.
Useful Links
The planning competition home page..