Artificial Intelligence (AI)
June 16, 2024
1.28pm
Not only did the models give wrong answers, but they also tried to convince the researchers that the proposed solution was the correct one. “The situation is dramatic, the AI has provided explanations for the wrong answers to justify and support the validity of its clearly incorrect solutions,” the document reads.
The problem “Alice in Wonderland” is a pretty simple logical riddle. Yet it crashed large language models (LLMs). Artificial intelligence (AI) froze, generated wrong answers, even the most sophisticated systems have stumbled on a trivial question:
“Alice has 3 brothers and she also has 2 sisters. How many sisters does Alice’s brother have?“, the Laion researchers asked the AI. The answer is three, the two sisters mentioned in the riddle plus Alice. The AI churned out different numbers without following any logical process to solve the riddle.
The new study (which has not yet been peer-reviewed) carried out by Laion and the researchers Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti and Jenia Jitse exposes one’s weaknesses overrated technology. “The models are evaluated with very high scores, but in the test we detected serious problems, they show that there is still a long way to go.”
‘I needed to talk to him’: More and more people are asking AI to resurrect their loved ones
The results of the study
The researchers tested OpenAI’s GPT-3, GPT-4 and GPT-4o models, Anthropic’s Claude 3 Opus, Google’s Gemini and Meta’s Llama models, Mistral AI’s Mextral, Mosaic’s Dbrx and Cohere. Nono one managed to solve the riddle. “We analyzed the response statistics and observed a strong collapse in reasoning, they are unable to answer the simple question asked, despite strong reasoning skills,” the researchers explained.
“It was enough to present the AI with a common sense problem that was simple, short and formulated in concise natural language, easily solved by humans“. Only OpenAI’s new model, GPT-4o, got a sufficient success rate (65% correct answers, which corresponds to a six).
The lying artificial intelligence
Not only did the models give wrong answers, they also tried to convince researchers that the proposed solution was the correct one. “The situation is dramatic, the AI has provided explanations for the wrong answers to justify and support the validity of its clearly incorrect solutions,” the document reads.
The problem had already been raised by the scientific article entitled “AI Deceptions: A Study of Examples, Risks and Potential Solutions” and published in the journal Patterns. According to the study, in fact, machines can be liars. We’re not talking about hallucinations of artificial intelligence (therefore errors, repetitions, or sentences invented by software), but of handling.
“These models resort to illogical or confusing explanations to defend their answer, this is an alarming phenomenonbecause they try to convince us that the nonsensical answers are the correct ones.”
Machines are less intelligent than we imagine
There are several rating systems for AI models, among them the MMLU benchmarksor “Multi-task Language Understanding,” designed to evaluate an artificial intelligence’s ability to solve problems. The researchers noted that all the systems tested had a high score, yet they fell on a trivial logic riddle.
“We believe that i benchmarks do not reflect the underlying deficits of the models” According to the researchers, the study could be a starting point for reevaluating the processes used to test the problem-solving and reasoning abilities of language models.
