How AI Knows Things No One Told It – Scientific American

No one yet knows how ChatGPT and its artificial intelligence cousins will transform the world, and one reason is that no one really knows what goes on inside them. Some of these systems abilities go far beyond what they were trained to doand even their inventors are baffled as to why. A growing number of tests suggest these AI systems develop internal models of the real world, much as our own brain does, though the machines technique is different.

Everything we want to do with them in order to make them better or safer or anything like that seems to me like a ridiculous thing to ask ourselves to do if we dont understand how they work, says Ellie Pavlick of Brown University, one of the researchers working to fill that explanatory void.

At one level, she and her colleagues understand GPT (short for generative pretrained transformer) and other large language models, or LLMs, perfectly well. The models rely on a machine-learning system called a neural network. Such networks have a structure modeled loosely after the connected neurons of the human brain. The code for these programs is relatively simple and fills just a few screens. It sets up an autocorrection algorithm, which chooses the most likely word to complete a passage based on laborious statistical analysis of hundreds of gigabytes of Internet text. Additional training ensures the system will present its results in the form of dialogue. In this sense, all it does is regurgitate what it learnedit is a stochastic parrot, in the words of Emily Bender, a linguist at the University of Washington. But LLMs have also managed to ace the bar exam, explain the Higgs boson in iambic pentameter, and make an attempt to break up their users marriage. Few had expected a fairly straightforward autocorrection algorithm to acquire such broad abilities.

That GPT and other AI systems perform tasks they were not trained to do, giving them emergent abilities, has surprised even researchers who have been generally skeptical about the hype over LLMs. I dont know how theyre doing it or if they could do it more generally the way humans dobut theyve challenged my views, says Melanie Mitchell, an AI researcher at the Santa Fe Institute.

It is certainly much more than a stochastic parrot, and it certainly builds some representation of the worldalthough I do not think that it is quite like how humans build an internal world model, says Yoshua Bengio, an AI researcher at the University of Montreal.

At a conference at New York University in March, philosopher Raphal Millire of Columbia University offered yet another jaw-dropping example of what LLMs can do. The models had already demonstrated the ability to write computer code, which is impressive but not too surprising because there is so much code out there on the Internet to mimic. Millire went a step further and showed that GPT can execute code, too, however. The philosopher typed in a program to calculate the 83rd number in the Fibonacci sequence. Its multistep reasoning of a very high degree, he says. And the bot nailed it. When Millire asked directly for the 83rd Fibonacci number, however, GPT got it wrong: this suggests the system wasnt just parroting the Internet. Rather it was performing its own calculations to reach the correct answer.

Although an LLM runs on a computer, it is not itself a computer. It lacks essential computational elements, such as working memory. In a tacit acknowledgement that GPT on its own should not be able to run code, its inventor, the tech company OpenAI, has since introduced a specialized plug-ina tool ChatGPT can use when answering a querythat allows it to do so. But that plug-in was not used in Millires demonstration. Instead he hypothesizes that the machine improvised a memory by harnessing its mechanisms for interpreting words according to their contexta situation similar to how nature repurposes existing capacities for new functions.

This impromptu ability demonstrates that LLMs develop an internal complexity that goes well beyond a shallow statistical analysis. Researchers are finding that these systems seem to achieve genuine understanding of what they have learned. In one study presented last week at the International Conference on Learning Representations (ICLR), doctoral student Kenneth Li of Harvard University and his AI researcher colleaguesAspen K. Hopkins of the Massachusetts Institute of Technology, David Bau of Northeastern University, and Fernanda Vigas, Hanspeter Pfister and Martin Wattenberg, all at Harvardspun up their own smaller copy of the GPT neural network so they could study its inner workings. They trained it on millions of matches of the board game Othello by feeding in long sequences of moves in text form. Their model became a nearly perfect player.

To study how the neural network encoded information, they adopted a technique that Bengio and Guillaume Alain, also at the University of Montreal, devised in 2016. They created a miniature probe network to analyze the main network layer by layer. Li compares this approach to neuroscience methods. This is similar to when we put an electrical probe into the human brain, he says.In the case of the AI, the probe showed that its neural activity matched the representation of an Othello game board, albeit in a convoluted form. To confirm this, the researchers ran the probe in reverse to implant information into the networkfor instance, flipping one of the games black marker pieces to a white one. Basically, we hack into the brain of these language models, Li says. The network adjusted its moves accordingly. The researchers concluded that it was playing Othello roughly like a human: by keeping a game board in its minds eye and using this model to evaluate moves. Li says he thinks the system learns this skill because it is the most parsimonious description of its training data. If you are given a whole lot of game scripts, trying to figure out the rule behind it is the best way to compress, he adds.

This ability to infer the structure of the outside world is not limited to simple game-playing moves; it also shows up in dialogue. Belinda Li (no relation to Kenneth Li), Maxwell Nye and Jacob Andreas, all at M.I.T., studied networks that played a text-based adventure game. They fed in sentences such as The key is in the treasure chest, followed by You take the key. Using a probe, they found that the networks encoded within themselves variables corresponding to chest and you, each with the property of possessing a key or not, and updated these variables sentence by sentence. The system had no independent way of knowing what a box or key is, yet it picked up the concepts it needed for this task. There is some representation of the state hidden inside of the model, Belinda Li says.

Researchers marvel at how much LLMs are able to learn from text. For example, Pavlick and her then Ph.D. student Roma Patel found that these networks absorb color descriptions from Internet text and construct internal representations of color. When they see the word red, they process it not just as an abstract symbol but as a concept that has certain relationship to maroon, crimson, fuchsia, rust, and so on. Demonstrating this was somewhat tricky. Instead of inserting a probe into a network, the researchers studied its response to a series of text prompts. To check whether it was merely echoing color relationships from online references, they tried misdirecting the system by telling it that red is in fact greenlike the old philosophical thought experiment in which one persons red is another persons green. Rather than parroting back an incorrect answer, the systems color evaluations changed appropriately in order to maintain the correct relations.

Picking up on the idea that in order to perform its autocorrection function, the system seeks the underlying logic of its training data, machine learning researcher Sbastien Bubeck of Microsoft Research suggests that the wider the range of the data, the more general the rules the system will discover. Maybe were seeing such a huge jump because we have reached a diversity of data, which is large enough that the only underlying principle to all of it is that intelligent beings produced them, he says. And so the only way to explain all of this data is [for the model] to become intelligent.

In addition to extracting the underlying meaning of language, LLMs are able to learn on the fly. In the AI field, the term learning is usually reserved for the computationally intensive process in which developers expose the neural network to gigabytes of data and tweak its internal connections. By the time you type a query into ChatGPT, the network should be fixed; unlike humans, it should not continue to learn. So it came as a surprise that LLMs do, in fact, learn from their users promptsan ability known as in-context learning. Its a different sort of learning that wasnt really understood to exist before, says Ben Goertzel, founder of the AI company SingularityNET.

One example of how an LLM learns comes from the way humans interact with chatbots such as ChatGPT. You can give the system examples of how you want it to respond, and it will obey. Its outputs are determined by the last several thousand words it has seen. What it does, given those words, is prescribed by its fixed internal connectionsbut the word sequence nonetheless offers some adaptability. Entire websites are devoted to jailbreak prompts that overcome the systems guardrailsrestrictions that stop the system from telling users how to make a pipe bomb, for exampletypically by directing the model to pretend to be a system without guardrails. Some people use jailbreaking for sketchy purposes, yet others deploy it to elicit more creative answers. It will answer scientific questions, I would say, better than if you just ask it directly, without the special jailbreak prompt, says William Hahn, co-director of the Machine Perception and Cognitive Robotics Laboratory at Florida Atlantic University. Its better at scholarship.

Another type of in-context learning happens via chain of thought prompting, which means asking the network to spell out each step of its reasoninga tactic that makes it do better at logic or arithmetic problems requiring multiple steps. (But one thing that made Millires example so surprising is that the network found the Fibonacci number without any such coaching.)

In 2022 a team at Google Research and the Swiss Federal Institute of Technology in ZurichJohannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joo Sacramento, Alexander Mordvintsev, Andrey Zhmoginov and Max Vladymyrovshowed that in-context learning follows the same basic computational procedure as standard learning, known as gradient descent. This procedure was not programmed; the system discovered it without help. It would need to be a learned skill, says Blaise Agera y Arcas, a vice president at Google Research. In fact, he thinks LLMs may have other latent abilities that no one has discovered yet. Every time we test for a new ability that we can quantify, we find it, he says.

Although LLMs have enough blind spots not to qualify as artificial general intelligence, or AGIthe term for a machine that attains the resourcefulness of animal brainsthese emergent abilities suggest to some researchers that tech companies are closer to AGI than even optimists had guessed. Theyre indirect evidence that we are probably not that far off from AGI, Goertzel said in March at a conference on deep learning at Florida Atlantic University. OpenAIs plug-ins have given ChatGPT a modular architecture a little like that of the human brain. Combining GPT-4 [the latest version of the LLM that powers ChatGPT] with various plug-ins might be a route toward a humanlike specialization of function, says M.I.T. researcher Anna Ivanova.

At the same time, though, researchers worry the window may be closing on their ability to study these systems. OpenAI has not divulged the details of how it designed and trained GPT-4, in part because it is locked in competition with Google and other companiesnot to mention other countries. Probably theres going to be less open research from industry, and things are going to be more siloed and organized around building products, says Dan Roberts, a theoretical physicist at M.I.T., who applies the techniques of his profession to understanding AI.

And this lack of transparency does not just harm researchers; it also hinders efforts to understand the social impacts of the rush to adopt AI technology. Transparency about these models is the most important thing to ensure safety, Mitchell says.

Go here to read the rest:

How AI Knows Things No One Told It - Scientific American