Illustration: Konstantin Inozemtsev/Getty Images

It was billed as an epochal event in humanitys history: For the first time a computer had proved itself to be as smart as a person. And befitting the occasion, the June story generated headlines all around the world. In reality, it was all a cheesy publicity stunt orchestrated by an artificial-intelligence buff in England. But there was an upside. Many of the worlds best-known AI programmers were so annoyed by the massive coverage, which they deemed entirely misguided, that they banded together. They intend to make sure the world is never fooled by false AI achievement again. The result is a daylong workshop, Beyond the Turing Test, where attendees aim to work out an alternative to the current test. The workshop will be held this coming Sunday in Austin at the annual convention of the Association for the Advancement of Artificial Intelligence.

Whether a particular computer program is intelligent, as opposed to simply being useful, is arguably an unanswerable question. But computer scientists have nonetheless been asking it ever since 1950, when Alan Turing wrote Computing Machinery and Intelligence and proposed his now-famous test. The test is like a chat session, except the human doesnt know if its a computer or a fellow person on the other end. A computer that can fool the human can be adjudged to be intelligent or, as Turing put it, thinking.

In the early days of AI, the test was considered by scientists to be too far beyond the current capabilities of computers to be worth worrying about. But then came chatbot programs. Without using anything that could be described as intelligence, they use key words and a few canned phrases well enough to persuade the unaware that theyre having a real conversation with a flesh-and-blood human. This genre of programs has been fooling some folks for decades, including this past summer, when Eugene Goostman, a chatbot program pretending to be a 13-year-old Ukrainian boy, persuaded a handful of people in England that it was a real boy. (The program was undoubtedly aided by peoples assumption that they were speaking to a disaffected teen with limited English language skills.)

The Goostman win was trumpeted widely in the media, to the enormous chagrin of legitimate researchers. Most of them just groused privately, but one, Gary F. Marcus, a New York University research psychologist, used his forum as a contributor to The New Yorker to raise the issue of whether Turings test had become too easy to game, and to urge the AI community to come up with a replacement. To his surprise, researchers from all over the world wrote in offering to help. Wed clearly touched a nerve, says Marcus.

The upshot: Marcus is cochairing the 25 January event, along with Francesca Rossi, of the University of Padova, in Italy, and Manuela Veloso of Carnegie-Mellon, in Pennsylvania.

Anatomy of an AI Test: Winograd schemas might be a better test of human-level artificial intelligence than the Turing test because they require reasoning about a broad body of knowledge. Each schema has four requirements.

1: Two parties (males, females, groups, objects) are mentioned in a sentence.

2. A pronoun or possessive adjective is used in the sentence to refer to one of the parties, but that word could also refer to the second party.

3. The question involves determining the referent of the pronoun or possessive adjective.

Original post:

Artificial-Intelligence Experts to Explore Turing Test Triathlon