LLM Hallucinations

Understanding, Detecting, and Mitigating Fabricated Outputs

What Are Hallucinations?

Hallucinations occur when Large Language Models generate content that is factually incorrect, nonsensical, or entirely fabricated—but presented with the same confidence as accurate information. This is one of the most significant challenges in deploying LLMs for production applications.

Unlike human errors, LLM hallucinations stem from how these models work: they predict statistically likely next tokens, not verified facts. The model has no internal "truth checker" and cannot distinguish between what it "knows" and what it's inventing.

⚠️ Critical Risk: Hallucinations can lead to legal liability, misinformation, damaged trust, and real-world harm—especially in healthcare, legal, and financial applications.

Risk & Scope Definition

Hallucination isn't just "wrong facts". In production, classify each endpoint by risk:

Low: Creative, brainstorming (hallucination tolerable).
Medium: Summarization (risk of missing details).
High: Financial, Legal, Healthcare, Automated Decisions. Here the standard must be Abstention + Verification.

Types of Hallucinations

Factual Hallucinations

Inventing facts, statistics, dates, or events that never happened.

"Einstein won the Nobel Prize in 1925 for relativity." (Wrong year, wrong reason)

Citation Hallucinations

Fabricating references, papers, URLs, or quotes that don't exist.

"According to Smith et al. (2023) in Nature..." (paper doesn't exist)

Identity Hallucinations

Attributing quotes, actions, or beliefs to real people incorrectly.

"As Mark Zuckerberg said in his famous 2019 TED talk..." (never happened)

Context Hallucinations

Misrepresenting or inventing details from provided context (especially in RAG).

Document says "revenue grew 10%" → Model says "revenue grew 15%"

Code Hallucinations

Inventing APIs, functions, or libraries that don't exist.

"Use pandas.auto_clean() to..." (function doesn't exist)

Reasoning Hallucinations

Confident but flawed logic, especially in multi-step problems.

"Since A implies B and B implies C, therefore C implies A" (invalid logic)

Why Do LLMs Hallucinate?

Statistical Pattern Matching

LLMs predict the most likely next token based on patterns—not truth. If "Einstein won the Nobel Prize in" is followed by "1921" more often than other years in training data, it outputs 1921. But patterns can be wrong or misremembered.

No Grounded Knowledge Base

Unlike a database, LLMs don't store facts explicitly. Knowledge is distributed across billions of parameters. There's no "lookup" mechanism—only generation.

Training Data Issues

Models learn from internet data containing errors, outdated info, contradictions, and fiction presented as fact. Garbage in, garbage out.

RLHF Side Effects

RLHF training rewards helpful, confident responses. This can make models more likely to fabricate answers rather than admit uncertainty.

Knowledge Cutoff

Models don't know about events after their training cutoff. When asked about recent events, they may confidently generate plausible-sounding but wrong answers.

Detecting Hallucinations

Method	How It Works	Pros / Cons
Self-Consistency	Generate multiple responses; flag disagreements	Easy to implement / Higher cost
LLM-as-a-Judge	Use another LLM to verify claims against sources	Good accuracy / Can hallucinate too
NLI (Natural Language Inference)	Fine-tuned models check if output is entailed by context	Fast / Limited to context-grounded claims
Knowledge Graph Verification	Cross-check entities and relations against KG	High precision / Requires structured KB
Web Search Verification	Search claims and compare results	Real-time data / Slow, complex
Confidence Calibration	Analyze token probabilities for uncertainty	No extra calls / Requires logprobs access

# Self-consistency example
from openai import OpenAI
client = OpenAI()

def check_consistency(prompt, n=3):
    responses = []
    for _ in range(n):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.7
        )
        responses.append(response.choices[0].message.content)
    
    # Compare responses for consistency
    # If they disagree significantly, flag as potential hallucination
    return responses, are_consistent(responses)

Approach: Generation + Audit

Transform the generation process into "Generation + Audit". Do not trust raw output.

1. Self-Consistency / Sampling

Generate multiple responses and measure divergence (SelfCheckGPT).

2. NLI / Entailment

Verify if the answer is logically supported by the context (Entailment).

3. Dedicated Detection

Models like Vectara HHEM that classify hallucination probability.

4. Atomic Fact Checking

Break response into atomic facts and validate each separately (FActScore).

Mitigation Strategies

0. Layer Zero: "Abstain by Design"

The most important rule: make the model not invent when evidence is missing.

Anchored Response: "Only answer using the provided sources."
Mandatory Citation: Require citation for every claim (claim-level).
Explicit Refusal: Separate "what I know" vs "what I am inferring".

1. Well-Executed Retrieval-Augmented Generation

RAG only reduces hallucination if retrieval, prompting, and validation are correct.

Deep RAG Checklist:

Retrieval: Hybrid (BM25 + Vector), Re-ranking (Cross-encoder), Query Rewriting.
Chunking: By semantic sections (not fixed size) to keep context.
Restrictive Prompt: "Use only context; if not there, say not found."
Post-Validation: Block response if groundedness score is low (RAGAS/TruLens).

2. Anchored Prompt Engineering

Explicitly instruct the model to acknowledge uncertainty. "Factual Mode" example:

"Answer ONLY based on the CONTEXT."

"For every claim, include the citation [#chunk_id]."

"If the answer is not in the context, say: Not found in the provided context."

"Do not invent names, numbers, dates, links, or sources."

3. Lower Temperature

Use temperature=0 or very low values for factual tasks. Higher temperatures increase creativity but also increase hallucination risk.

4. Chain-of-Thought Verification

Ask the model to show its reasoning. Flawed reasoning is easier to catch than confident but wrong answers. Add "verify each step" instructions.

5. Constrained Generation

Reduce the model's freedom. The freer it is, the higher the risk.

Low Temperature: 0-0.2 for factual tasks.
Structured Format: Require JSON with fields (`answer`, `claims`, `citations`, `confidence`).
Guided Decoding: Restrict output to a specific schema.
Tool Calling: Force the model to query APIs/DBs instead of inventing.

6. Runtime Guardrails

Use frameworks like NVIDIA NeMo Guardrails or Guardrails AI as an output "firewall". Golden rule: the guardrail must be able to block, ask for more context, or force abstention if inconsistency is detected.

7. Human-in-the-Loop (HITL)

For high-stakes applications, flag uncertain or critical responses for human review. Let humans verify before publishing or acting on LLM output.

Continuous Evaluation & Metrics

Without metrics, you only "think" you improved. Build an evaluation suite with a Golden Dataset (real questions, adversarial, and out-of-scope).

Metric	Description	Tools
Faithfulness	Factual consistency with the retrieved context.	RAGAS
Groundedness	Relationship between answer and context (Context Relevance, Answer Relevance).	TruLens (RAG Triad)
Factual Precision	Precision by "atomic facts".	FActScore
Abstention Rate	How often it correctly says "I don't know".	Manual / Custom
Citation Precision	If the citation actually supports the statement.	Manual / NLI

Tools & Frameworks

RAGAS: RAG evaluation framework with faithfulness and answer relevancy metrics
TruLens: LLM app evaluation with groundedness and hallucination detection
Guardrails AI: Output validation with fact-checking rails
NeMo Guardrails: NVIDIA's framework for LLM safety and grounding
Vectara: RAG platform with built-in hallucination detection (HHEM)
LangSmith: Tracing and evaluation for debugging hallucinations
OpenAI Evals: Framework for custom hallucination benchmarks

Advanced Strategies (Enterprise Level)

A. Answerability Gating

Before answering, evaluate: "is there sufficient evidence in retrieved docs?". If not, the bot must ask for details, reply "not found", or trigger additional search (web/KB).

B. Multi-stage RAG

Retrieve → Summarize evidence with citations → Generate final answer only from cited summary. This reduces the model's "leakage" to its pre-trained knowledge.

C. Automated Reasoning

For formal domains (compliance, law), use rule-based verification and logic (automated reasoning checks) instead of just free text.

Recommended Architecture Blueprint

High Stability Pipeline

1 Classify Intent & Risk (Factual vs Creative; High vs Low Risk)
2 RAG Retrieval (Hybrid + Reranker)
3 Constrained Generation (Low Temp + Schema + Mandatory Citations)
4 Verification (Claims & Groundedness) (Extract claims → Check RAGAS/NLI)
5 Decision (Pass → Return | Fail → Abstain/Feedback)

Other Best Practices

Log everything: Store prompts, contexts, outputs, scores, and decisions for observability.
Add disclaimers: For user-facing apps, clearly indicate that AI outputs should be verified.
Implement feedback loops: Allow users to report inaccurate responses.

Test Your Knowledge

Score 8/10 or higher to pass