Key takeaway: A hallucination is a confident statement with no support. Where the claim type is constrained, verification is a lookup rather than a judgement.
Why Confidence Scores Do Not Help
The intuitive approach is to read token probabilities and flag low-confidence output. This fails because hallucinations are frequently high-confidence. The model is not uncertain when it invents a plausible citation — the fabricated text is exactly what its training distribution suggests should come next.
Confidence correlates with fluency, not with truth. A well-formed invented API method scores higher than an awkwardly phrased correct one.
Checks That Actually Work
Verify every claim type that has a source of truth.
If the model names a function, check it exists in the API surface. If it cites a document, confirm the identifier resolves. If it produces a URL, request it. If it states a number, compare against the record. These checks are cheap, deterministic and catch the most damaging category of error.
Require and validate grounding spans.
In a retrieval system, ask the model to return the supporting quote alongside each claim, then verify that quote appears verbatim in the retrieved context. A fabricated claim usually comes with a fabricated or approximate quote, and exact string matching catches it.
{
"answer": "Enterprise tier allows 10,000 requests per minute.",
"support": "Enterprise: 10,000 req/min",
"source_id": "doc_442"
}If support is not a substring of document 442, reject the answer rather than displaying it.
Sample and compare.
Generate the same answer three times at moderate temperature. Facts the model actually knows stay consistent. Fabrications vary, because they are drawn from a flat region of the distribution. Disagreement across samples is a strong hallucination signal and requires no external knowledge.
Constrain the output space.
Where the valid answers form a known set, use structured generation or validate against the set. A model cannot hallucinate a category that the schema does not permit.
Ranking the Options
| Method | Cost | Catches |
|---|---|---|
| Existence checks against real APIs | Very low | Invented identifiers |
| Grounding span verification | Very low | Unsupported claims |
| Multi-sample consistency | 3× inference | Uncertain fabrication |
| Schema constraint | Negligible | Out-of-domain answers |
| LLM-as-judge | 1 extra call | Reasoning errors |
| Human review | High | Everything |
The first four require no additional model call and cover the majority of production hallucinations. LLM judges are useful for subtler reasoning problems but are themselves fallible and cannot be relied on as the only gate.
Designing for Refusal
The most valuable single behaviour is a model willing to say it does not know. Prompt explicitly for that, provide examples of appropriate refusal, and treat an unnecessary refusal as a far cheaper error than a confident fabrication.
Then measure both rates. A system that never refuses is hallucinating; one that always refuses is useless. The ratio is the tuning target.
The Bottom Line
Verify claims that have a source of truth, require verbatim grounding spans and check them by string match, and use multi-sample agreement where nothing external can confirm. Reserve model-based judging for what deterministic checks cannot reach.
