What hallucination is, and why it cannot be prompted away
Hallucination is plausible but wrong output: a function name that does not exist, an invented paper, a miscalculated total. It follows from how these models work — they continue the most likely next token, they do not look facts up in a database. So hallucination cannot be eliminated by prompting; only process can keep it from reaching production.
Four high-risk scenarios
| Scenario | Typical failure | Defence |
|---|---|---|
| API and function names | Invented parameters and return values | Official docs, IDE navigation, compile |
| Numbers and arithmetic | Wrong totals, percentages, groupings | Let tools compute; never let it do mental maths |
| Citations and sources | Fabricated papers, links, clause numbers | Demand checkable links and open each one |
| "I'm sure this feature exists" | Attributing library A's ability to library B | Write a 20-line minimal repro and run it |
A six-step checklist
- Separate fact from inference: require "unknown when unsure" and a stated basis for every claim;
- Demand checkable sources: doc links or file paths for every key conclusion;
- Use tools instead of mental maths: arithmetic, encoding, hashing and file checks belong to deterministic tools;
- Minimal reproduction: for any API or dependency claim, run the smallest possible example;
- Automated gatekeeping: validate structured output against a schema before business logic;
- Human sign-off: anything externally published, billed or regulated needs a person.
Three reusable prompt fragments
// 1. Force uncertainty labels
Label each claim [Confirmed] / [Inferred] with its basis;
write [Unknown] when you have none
// 2. Forbid invented sources
Only cite links you can provide; if you cannot, say
"I could not find a reliable source"
// 3. Hand maths to tools
Do not state the result - give the formula and inputs,
and I will compute it with a toolReal-world scenarios: three high-risk hallucinations
- Fabricated citations and links: plausible-looking titles, DOIs and URLs that do not exist — open every one to verify.
- Outdated or reversed facts: the model may state an old version confidently; check versions and numbers against the source.
- Convincing-looking math: full steps but a wrong intermediate value — re-check key calculations with a tool or code.
Common questions
Does lowering temperature stop hallucination? It reduces randomness, not error — wrong content can be produced very consistently. Is self-review useful? Somewhat (consistency checking), but it can repeat the same mistake, so external verification is still required. Does RAG make it safe? It lowers invention rates; retrieved fragments may be outdated or irrelevant, so sources still need checking.
Build verification into the process, not into willpower
- Structure the output: ask for JSON and validate it against a schema before business logic sees it — that keeps "almost right" out of the system;
- Require sources for key claims: no unsourced conclusion goes into docs, code comments or external replies;
- Hand all computation to tools: the model describes steps and formulas; it does not state final numbers;
- Keep a human sign-off list: anything published, billed, regulated or security-relevant gets checked line by line;
- Collect failure samples: turn each discovered hallucination into a regression case so you can tell whether prompt and process changes actually help.
Try them: hashing, JSON validation
Cross-checking techniques
- Ask again, differently: rephrase the question or invert it — contradictory answers are a red flag;
- Demand sources: require it to point at the passage it relies on; anything unsourced is unverified;
- Split into checkable units: break a conclusion into independently verifiable facts — far more reliable than judging the whole by plausibility;
- Scrutinise three categories: numbers and dates, API and function signatures, and legal or standards clauses — most often fabricated and most costly when wrong.