Call for Papers (PDF)
Many subfields of artificial intelligence and robotics regularly host
competitions, such as the RoboCup soccer and RoboCup Rescue
competitions (robotics), the SAT competition (boolean satisfiability),
the International Planning Competition (action planning), the Trading
Agent Competition (agents), the CADE ATP System Competition (automated
theorem proving), the Annual Reinforcement Learning Competition
(reinforcement learning), the Diagnostic Competition (model-based diagnosis),
or the CSP solver competition (constraints).
Competitions impact research communities in many ways, including a
scientific, engineering and community dimension. Scientifically, they
offer a way to evaluate the state of the art of a subfield by
providing a common benchmark on which different approaches to a
problem can be compared. From the engineering perspective, they
help technology in an area to mature by requiring development of
systems that work robustly on unseen problems or by promoting the
development of tools or reusable system components for the problem
addressed by the competition. From the community perspective, they
inspire discussion and attract publicity for a field and help enroll
young researchers in a research community.
There are many subfields of artificial intelligence and robotics in
which competitions have had a clear influence on the research
landscape in past years:
- In robotics, the RoboCup competitions (originally on robotic soccer,
recently also in search and rescue scenarios) have attracted huge
publicity and inspired a large number of researchers to work on its
challenges. RoboCup has effectively evolved into an own subfield
where research activity is to a large extent guided by the
requirements defined by the competition. More recently, the DARPA
Grand Challenge has spurned a flurry of research activity on
autonomous navigation in large outdoor areas, leading to impressive
improvements of the state of the art.
- In satisfiability testing, the SAT competitions have provided a
continuous challenge for solvers that has inspired significant
algorithmic innovations for SAT solvers as well as huge improvements
in implementation quality (e.g., low-level performance).
- In classical planning, the International Planning Competitions have
focused the research community on a common representation language,
PDDL, and a set of common benchmarks which have greatly helped
comparing different classical planning systems to each other. They
have also led to a huge increase in scalability of planning systems
on a wide range of problem domains.
But competitions haven't had the same degree of impact in all
subfields of artificial intelligence or robotics. In model-based
diagnosis for instance the community has just started converging on a
generally accepted way of evaluating and comparing different approaches or
technologies. Some researchers argue that the missing confidence in
the methods used to evaluate approaches has been an obstacle to
progress in this area.
Despite the potential advantages resulting from competitions, they
have been a source of controversy in many subfields of
artificial intelligence. Whereas supporters believe that competitions
accelerate research, opponents argue that they often focus research on
synthetic problems or preclude research directions that are less
aligned with current competitions.
We believe that the methods used to evaluate and compare research have
strong implications on future research directions and therefore need
to be well designed. Once communities have accepted regular
competitions, it can be difficult to create new directions in
research. Another important aspect is the question of how competition
should evolve as research evolves. Therefore a careful design as well
as the actively guided evolution of competitions is essential for its
success in the field as well as for the success of the field.