AI & TECHNOLOGY / PRACTICAL GUIDE
AI Hallucinations: Definition and How to Check Them
An AI hallucination, called confabulation in NIST guidance, is generated content that is false, erroneous or inconsistent while still being presented with total confidence. It comes from how these systems produce statistically likely text, not from anything like human belief or an intent to deceive — which is exactly why checking it needs a method, not a gut feeling.
In this guide
- Confidence is a style signal, not a fact-check
- The case: one generated answer, five claims
- What the check actually turned up
- Why this happens
- Five ways a wrong answer gets made
- A verification routine you can actually run
- Triage, document, escalate
- Set a stopping rule
- What to do next
- Sources and further reading
Confidence is a style signal, not a fact-check
Grammar, specific-sounding numbers and even a neatly formatted citation can make an answer look authoritative regardless of whether the underlying claim is true. Presentation and evidence come from two separate processes, and only one of them is actually checkable.
The case: one generated answer, five claims
Fictional worked example: Northfield Robotics, its people, figures and verification outcomes below are invented to demonstrate the checking method.
Say you ask an AI assistant to summarize the background of a company we'll call Northfield Robotics, and it returns a tidy five-sentence bio. Before repeating any of it, break the paragraph into individual, checkable claims — one row per claim, and one plan for how you'd confirm each.
| Claim from the generated answer | How you would check it |
|---|---|
| "Founded in 2016 by Elena Cho." | Company filings, an official about page or a business registry naming a founder and date. |
| "Raised $40 million in a 2021 Series B round." | A funding database or press release naming the round, amount and investors. |
| "Flagship product is the R-9 warehouse arm." | The company's own product page or an independent trade-publication review. |
| "Has never had a product recall." | A safety regulator's recall database — though an empty search only means none was found, not proof none happened. |
| "Headquartered in Austin, Texas." | Two official sources you'd expect to agree, such as a filing and the company's own contact page. |
What the check actually turned up
- Founding year and founder — supported: Matched an independent filing and the company's own materials, so this claim held up.
- Funding figure — unsupported: No such round appears in any funding record. The number was plausible-sounding and simply invented to fill a gap in what the model actually knew.
- Headquarters — uncertain: One official source said Austin; an older one said Dallas. The honest verdict is "uncertain, pending a current source," not a coin-flip guess at which is right.
Three different verdicts from one short paragraph — supported, unsupported and uncertain — is a normal outcome, not a rare one. The point of the exercise is catching that spread before any of it gets repeated as settled fact.
Why this happens
A generative model produces the statistically likely next piece of text given everything it has seen and the prompt it was given. When a real answer is thin, unclear or missing from its training, a fluent, wrong answer is often just as easy for it to produce as a correct one — the process behind it doesn't distinguish the two the way a person checking a fact would.
Terminology: NIST guidance now uses the term confabulation rather than hallucination for exactly this reason: it names how the content was generated, not anything resembling human perception, memory or belief.
Five ways a wrong answer gets made
"The model made it up" is true but not useful. Different mechanisms produce different kinds of error, and knowing which one you are probably looking at tells you where to check first.
- Thin training data on a niche topic: A small company, a local regulation or an obscure paper gives the model little to draw on, so it fills the gap with whatever is typical. Example: the Northfield funding round above, a normal-sounding number attached to a company it barely knew.
- Stale data in the present tense: Training stops at a cutoff, but the answer reads as if it were written this morning. Example: "The CEO is Elena Cho" when she stepped down last year and the model never saw the announcement.
- A leading prompt: Ask "Why did Northfield recall the R-9 in 2022?" and the model will often explain a recall that never happened, because the question already asserted it. Example: any prompt beginning "Explain why" about an event you have not confirmed.
- Long context, lost details: Paste a forty-page document and ask about page three, and the answer may blend page three with page thirty. Example: a summary of a long transcript that attributes a quote to the wrong speaker.
- Citations that look real and are not: The model has seen millions of references and can produce the shape of one perfectly. Example: "Smith and Patel (2019), Journal of Robotics Safety, 14(2)," where the journal may exist but the article does not.
The last one defeats the usual instinct, because a citation feels like evidence. Treat it as one more claim: open the source rather than noting that one was named.
A verification routine you can actually run
- Split the answer into claims: One factual statement per line — names, dates, numbers and anything presented as settled.
- Find an independent source for each one: Not another AI-generated summary, and not a source that could have copied the same original error.
- Check the exact wording: A source can support the general topic while contradicting the specific number or date given.
- Note the source's date: A true claim from 2019 can be a false one today.
- Mark the verdict: Supported, unsupported or still uncertain — and keep the uncertain ones labeled that way instead of rounding them up to true.
Triage, document, escalate
The routine above handles one answer. Three additions keep regular checking from becoming either a full-time job or a formality. First, triage by what is riding on the claim, because not every line deserves the same effort.
| The claim is used for | Scrutiny it needs | Good enough looks like |
|---|---|---|
| Your own curiosity or a rough draft | Light | One quick independent source, or an honest note that you never checked |
| Something you will repeat to other people | Standard | One independent source per claim, exact wording confirmed, date noted |
| A decision involving money, health, law or safety | Full | Two independent sources that agree, or a qualified person, before anyone acts |
Second, document what you checked, briefly. One line per claim is plenty: the claim, the source used, the date you looked, the verdict. "Funding round: searched two funding databases on 14 May, nothing found, unsupported." Six months later that line tells you whether the check is still current, and it lets someone else see what was actually confirmed instead of taking your word for it.
Third, escalate instead of resolving it yourself when the claim sits in a domain where being wrong has real consequences and you are not qualified to judge the sources. A drug interaction, a tax treatment, a contract clause: the right move is not a third search but handing the specific claim and your notes to a pharmacist, an accountant or a lawyer, with the honest label "an AI tool said this and I could not confirm it." That handoff is a verification step, not a failure of one.
Set a stopping rule
If a claim is consequential and you cannot trace it to a suitable source or a qualified person, the answer is not "probably fine" — it is "not yet checked." Record the uncertainty and act only on what you have actually confirmed.
What to do next
- Pick one AI-generated answer already sitting in a chat history and split it into individual claims before trusting any of it.
- For each claim, find one independent source that isn't itself another AI summary, and check the exact wording, not just the general topic.
- Label every claim supported, unsupported or uncertain, and only share or act on the ones marked supported.