AI can be a little delusional sometimes; you can ask it a completely normal question and get an answer that sounds so confident you’d probably assume it knows what it’s talking about, only to realise later that it has made half of it up.
That’s basically what an AI hallucination is: information that sounds convincing but isn’t actually true. A made-up fact in a casual chat might get a quick “lol, okay” before you move on, but the consequences can be very different when that information gets copied, shared or treated as a fact by someone else. The problem isn’t only that AI can get things wrong; it’s that it can sound completely sure when it does.
When An AI Mistake Becomes A Real-World Problem
AI hallucinations don’t always stay inside the original conversation; so once an answer has been copied somewhere else, it can be separated from the AI that produced it and passed on as if it came from a normal source.
The US National Institute of Standards and Technology (NIST) describes these errors as “confabulations”, where generative AI produces false or erroneous information. Language models generate likely sequences of text rather than checking every statement against reality first, which means an answer can sound pretty certain even when the information behind it is completely wrong.
An AI system can produce a specific statistic, legal citation or explanation without giving the reader any obvious indication that it needs checking. Sometimes it can even provide a convincing-looking source for information that doesn’t even exist, which makes checking the information a bit more important rather than just accepting it as it is.
When AI Hallucinations Enter Serious Situations
In Australia, Deloitte was paid around AU$440,000 to produce a report for the government looking at its welfare compliance system. The report was later found to have incorrect and nonexistent references, after University of Sydney academic Dr Christopher Rudge flagged problems with some of the citations; Deloitte then agreed to repay the final instalment of the contract.
In 2026, Reuters reported that New Mexico lawyer Stephen Aarons was fined $5,000 after submitting a murder appeal brief containing fabricated police testimony and fictitious witnesses generated by ChatGPT. Aarons had used ChatGPT to summarise trial proceedings and said he hadn’t realised how easily the system could make things up.
Healthcare is another area where a wrong answer can cause problems, especially when AI is being used to turn speech into written records. A 2024 study of OpenAI’s Whisper speech-to-text system found hallucinated text in some of the audio segments tested, including phrases and sentences that were not present in the original recordings. According to the Pulitzer Center, more than 30,000 clinicians across 40 health systems had started using an AI transcription tool similar to Whisper to transcribe doctor-patient conversations. If an AI-generated transcription contains something that was never actually said, that can become a problem when it’s being used as part of a patient’s medical documentation.
Then there is the Chinese ship incident reported by TechCrunch – according to its report, a US military intelligence assessment produced with help from an AI chatbot incorrectly claimed that a Chinese vessel in the Middle East was carrying components for a nuclear weapons programme. Military aircraft were reportedly already in the air and an armed operation was being prepared before officials discovered that it had been hallucinated and called it off.
Why Are Hallucinations So Difficult To Spot?
Part of the problem is how generative AI produces language – the US National Institute of Standards and Technology (NIST) explains that large language models generate likely sequences of text based on patterns in their training data, which can produce accurate information but can also result in factual errors or contradictions.
There is no automatic warning attached to the incorrect information either. An AI system can present a genuine fact and an invented one in almost exactly the same tone, with the same neat formatting and apparent confidence.
A legal document, consultancy report or intelligence briefing can make an AI-generated claim seem more credible simply because it’s sitting alongside other information that we expect to be accurate. If someone assumes it has already been checked, the mistake can easily get passed along without anyone even stopping to question it.
Can Technology Catch AI Hallucinations?
There are technical ways to reduce the risk; Retrieval-Augmented Generation, or RAG, allows an AI system to retrieve information from an external knowledge base before generating its response, helping ground the answer in specific source material.
Researchers are also developing systems that can detect potentially hallucinated information and assess whether an AI-generated claim is supported by its sources. These methods can help catch errors, but they don’t guarantee that every incorrect answer will be identified.
Having another AI check the first one could sound like a solution, but someone still needs to look at the result and decide if it actually makes sense, especially when the answer could affect a real decision. AI can help with that process, but it probably shouldn’t be the only thing checking its own homework.
What Happens When AI Gets Used Without Enough Oversight?
The problem with an AI hallucination isn’t always the wrong answer, but what happens after someone believes it. A made-up detail can be copied into a document, passed on to someone else or used as part of a really big decision before anyone realises it was wrong.
AI can be useful without getting everything right, but once people stop checking what it produces, a confident-sounding mistake can start being treated like fact – even when it never deserved that level of trust in the first place.
