On April 2, the World Health Organization (WHO) launched its new chatbot, SARAH (Smart AI Resource Assistant for Health), with great expectations. This virtual assistant, based on GPT-3.5 technology, was designed to offer health advice in eight different languages.
The idea was to provide an accessible and useful service to millions of people around the world, helping them make informed decisions about their well-being. However, what promised to be a revolutionary tool quickly became a source of problems.
Just days after its release, SARAH was criticized for providing incorrect information. In one particular case, he even invented names and addresses of non-existent clinics in San Francisco, which could have had serious consequences if not detected in time.
This incident is just one more in the growing list of AI-powered chatbot failures. In the past, other systems such as Meta’s Galactica and Air Canada’s chatbot have also suffered similar problems.
But why does this happen? Why do these artificial intelligences, designed to help us, end up inventing answers that have no basis in reality?
All AIs are amazed
What happened to SARAH is a phenomenon known as “AI hallucination.” In the field of AI, a hallucination refers to an AI-generated response that contains false or misleading information presented as fact.
Although we typically associate the word ‘hallucination’ with human experiences, in the context of AI, it has been used as an analogy. The key difference is that an AI’s hallucination is related to incorrect responses or beliefs, rather than perceptual or sensory experiences.
For example, a chatbot powered by large language models (LLMs), such as ChatGPT, can include random, plausible-sounding falsehoods within the generated one.
According to a recent study by Cornell researchers, all generative AI models blow your mind. However, not everyone does so at the same rate, and the veracity of their claims depends on the sources of information to which they have been exposed.
The study evaluated several popular models, including GPT-4o, Llama 3 70B, Mixtral 8x22B, among others. Surprisingly, even the largest and most advanced models, such as GPT-4o, only generate hallucination-free text about 35% of the time.
How and why do AI hallucinations occur?
Hallucinations occur when an AI model, such as a neural network, produces results that do not directly relate to the inputs provided. These results can be patterns, objects, or even text that we didn’t expect or that don’t make sense.
Current AIs (based on the transformer architecture) are prone to hallucinations due to the way they learn from data. During training, models try to capture patterns in the data, but sometimes they extrapolate or fill in gaps incorrectly.
These are some factors that can contribute to hallucinations:
- Errors in training data: If the data contains errors or biases, the model can learn to replicate them. Hallucinations can arise when the model generalizes incorrectly from incorrect examples.
- Very complex models: Deep neural networks have more parameters and therefore more opportunities to freak out. Complexity increases the likelihood that the model will generate incorrect information.
- Faulty generalization: Models can learn patterns that do not apply universally. This can lead to hallucinations when faced with unusual situations or situations outside of their training domain.
- Overfitting: If a model fits the training data too much, it may memorize specific examples, rather than learning general concepts. This can cause hallucinations when encountering new data.
Consequences of hallucinations
AI hallucinations can have serious implications, especially in fields where precision is crucial, such as medicine. In SARAH’s case, the creation of fictitious clinics could have led to poor decisions in seeking medical care.
In other cases, these hallucinations can perpetuate biases and prejudices present in the training data. For example, a model could generate discriminatory or stereotypical statements, as occurred in February with the generation of images in Google Gemini.
Finally, when users realize that an AI model is hallucinating, they may lose confidence in its ability to provide accurate answers.
The underlying problem is that, as these models become more sophisticated and their responses more credible, it is easier for users to blindly trust them.
This can lead to the spread of false information, as we have already seen in the case of chatbots that invent court quotes or scientific articles.
Do AI hallucinations have any use?
As we have seen, AI hallucinations are often considered a problem, but they also have potential applications in creative and exploratory fields. For example, the hallucinatory capabilities of AIs can be a valuable tool in art and design.
AI models can generate images that defy logic and reality, opening up new creative possibilities. These images can inspire artists and designers to explore unconventional territories.
If used to generate music, hallucinating AI can create original melodies and unusual harmonies. In the film and video game industry, hallucinations can lead to surprising and unique visual effects.
They may also be useful in data visualization, where AI’s ability to see non-obvious connections could provide new insights.
Mitigating the problem of hallucinations
To minimize hallucinations, several measures can be taken. One of the most fundamental methods to reduce hallucinations is to train models with well-structured, diverse, and high-quality data sets.
This includes data that reflects a wide range of contexts and is free of biases that could lead the model to generate incorrect responses.
Implementing constraints that limit the range of possible responses is another effective technique. This can be achieved by using filtering tools and defining clear probabilistic thresholds, which the model must meet before generating a response.
You can also use the “chain of thought” technique (Chain-of-Thought Prompting). This technique invites the model to decompose its reasoning step by step, allowing it to verify the logic of its answers as it generates them.
Lastly, and perhaps most importantly; Human oversight remains crucial to ensuring the accuracy of AI systems. Involving human experts to review and validate model outputs is a last line of defense against hallucinations.
SARAH’s case is a reminder that, although artificial intelligence has advanced enormously, it remains imperfect. AI hallucinations not only reveal the current limitations of the technology, but also underline the importance of a cautious and critical approach in its implementation.
This post is also available in: