Hallucinations & LLM Limitations

The precise definition matters. A hallucination is output that is presented as true but is either factually wrong or not supported by the available evidence. If the user provides a contract and the model says the termination clause requires 90 days when the document says 30 days, that is an intrinsic hallucination. If the model adds that the contract was reviewed by a named law firm when no source says that, it is extrinsic.

Fluency makes hallucinations hard to catch. The model may use correct grammar, domain vocabulary, and confident phrasing while the underlying claim is false. Production systems must therefore evaluate the evidence behind the answer, not the surface quality of the prose.

A language model learns statistical regularities from training data and uses the prompt plus previous tokens to predict a distribution over the next token. That objective rewards plausible continuation, not truth verification. Unless the system adds retrieval, tools, or explicit checks, the model has no built-in ground truth source at generation time.

Hallucinations become more likely when the prompt asks for obscure facts, the answer depends on recent events, the model has seen conflicting examples, or the conversation pressures it to provide an answer anyway. Higher temperature and loose instructions can increase variety, which can also increase unsupported claims.

Intrinsic hallucination contradicts the source the model was supposed to use. In summarization, that means the summary says the report revenue grew when the report says it declined. In question answering, it means the answer conflicts with the retrieved passage.

Extrinsic hallucination adds information that is not present in any source. It may be true in the outside world, but it is still unsupported by the evidence available to the system. This distinction is important for evaluation because intrinsic errors can be caught by contradiction checks, while extrinsic errors require source attribution and evidence coverage checks.

LLMs also have limitations that are not exactly hallucinations. They may have stale knowledge because training data ends at a cutoff. They may count tokens, characters, or items poorly because generation is not exact symbolic computation. They may be sensitive to small prompt changes, lose information beyond the context window, reproduce training biases, or agree with a user even when the user is wrong.

These limitations share a product lesson: the model is a probabilistic component, not an oracle. Treat it as one part of a larger system that can retrieve evidence, call deterministic tools, apply policies, and escalate uncertain cases.

There is no single switch that removes hallucinations. Reliable products combine grounding through retrieval, citations that expose evidence, tool use for fresh facts and math, constrained output formats, verification passes, conservative decoding, human-in-the-loop review, and continuous evaluation.

The design goal is not to force the model to always answer. The design goal is to make the product truthful about what it knows, transparent about sources, and safe when evidence is missing. A good system can answer, ask for clarification, call a tool, or say I do not know depending on the situation.

Intrinsic hallucinations are especially damaging in summarization and RAG because the product claims to be grounded in a source. Examples include reversing a legal obligation, changing a date, inventing a conclusion from a report, or answering with a fact that contradicts the retrieved passage.

The mitigation is faithfulness checking: compare answer claims against source spans. Strong systems require every material claim to map to evidence, and they block or revise answers when the evidence says something different.

Extrinsic hallucinations often look helpful because they add context. The model may invent a customer name, cite a paper that does not exist, include a plausible URL, or add a best practice that was never in the source. In an open web answer, the extra claim may even be true, but the product cannot justify it from the evidence it retrieved.

Source attribution is the key control. If a claim cannot be attached to a source, the answer should either remove it, label it as general background, or ask permission to search for more evidence.

LLMs can explain arithmetic patterns but still fail at exact computation. Counting words, summing invoice rows, comparing dates, and validating formulas should be delegated to deterministic tools. The model can plan the calculation and explain the result, but the calculation itself should come from code, SQL, or a calculator.

This also applies to structured constraints. If the output must be valid JSON, valid SQL, or a policy-compliant decision, use schemas, parsers, validators, and retry loops rather than trusting the first generation.

Small prompt changes can move the model to a different region of its learned distribution. A leading question can make it agree with a false premise. A higher temperature can produce different answers on repeated runs. These behaviors are not bugs in the usual software sense; they are properties of probabilistic generation.

Mitigations include stable prompt templates, explicit refusal and uncertainty policies, adversarial test sets, lower temperature for factual tasks, and self-consistency checks where multiple attempts must converge before the system trusts the answer.

A reliable AI product needs a first-class failure path. When evidence is missing, the system should say I do not know, ask for more context, route to search, escalate to a human, or explain what it can answer safely. This is a product feature, not a weakness.

Graceful failure also means avoiding overconfident UI. Do not hide uncertainty behind polished prose. Show sources, caveats, and next actions in a way users can understand.

You cannot prove reliability by reading a few impressive examples. Build evaluation sets that include groundedness checks, factuality questions, stale-knowledge cases, adversarial prompts, unsupported-citation traps, and regression examples from production incidents.

Track metrics such as faithfulness to source, citation precision, answer correctness, refusal correctness, tool-call success, and human escalation rate. Red-team the system before launch and after every major model, prompt, or retrieval change.

Guardrails should operate before, during, and after generation. Input filters can detect unsafe or out-of-scope requests. Prompt and policy constraints can limit what the model is allowed to claim. Output validators can require citations, schema validity, and policy compliance.

For sensitive domains, human-in-the-loop review is often mandatory. The system should make review efficient by showing evidence, model reasoning summaries, flagged unsupported claims, and a clear audit trail.

This toy checker illustrates the core idea behind groundedness evaluation. A production evaluator would use stronger claim extraction and semantic matching, but the principle is the same: important answer claims must be supported by evidence.

This example shows a simple reliability rule: let deterministic code compute totals, then let the model explain the result using that trusted output. In a real system the tool might be SQL, search, a calculator, or a business API.