The Turing test, now over 60 years old, has long served as a highly visible, public signpost for research in artificial intelligence. It is also highly game-able, and arguably in desperate need for a refresh.
The purpose of this workshop, modeled on a set of early meetings that helped shape the annual RoboCup competitions, is to seek community input. More precisely, at this workshop, our goal is to craft a replacement, an annual or bi-annual Turing Championship, that might consist of 3-5 different challenging tasks, with bragging rights given to the first programs to achieve human-level performance in each task.
With the help of the workshop participants, we envision the support and definition of at least two events. The first, recently sponsored by Nuance, will be the Winograd Schema Challenge, proposed by Hector Levesque, which tests the ability of machines to resolve linguistic antecedents in contexts in which common-sense knowledge is critical. The second, recently suggested by the workshop cochair, Gary Marcus, in an essay in the New Yorker, will focus on the comprehension of novel materials, such as videos, texts, photos, and podcasts. As an example, Marcus suggested a competition in which programs might be asked to watch "any arbitrary TV program or YouTube video and answer questions about its content — Why did Russia invade Crimea? or Why did Walter White consider taking a hit out on Jessie?" Several leading researchers, including Guruduth Banavar, Ned Block, Ernest Davis, Oren Etzioni, Ken Forbus, Hiroaki Kitano, Danica Kragic, Leora Morgenstein, Charles Ortiz, Stuart Shieber, Moshe Vardi, and Patrick Winston have agreed to be in the advisory board of this initiative.
Our hope is that an annual (or semiannual) Turing Championship can simultaneously generate public interest and serve as benchmarks that guide important and foundational AI research.
Gary Marcus, Francesca Rossi, Manuela Veloso (workshop organizers)