RAG

What is RAG?

RAG (Retrieval-Augmented Generation) is an architecture that combines knowledge base search capabilities with LLM text generation. Instead of relying only on the model's "frozen" knowledge, RAG allows the LLM to access up-to-date and domain-specific information.

RAG solves the "hallucination" problem by grounding responses in real, verifiable documents.

RAG Architecture

Query User Question

Embed Embedding

Retrieve Vector Search

Augment Context Injection

Generate LLM Response

Key Components

Embeddings

Transform text into numerical vectors that capture semantic meaning, enabling similarity comparison.

Vector Database

Stores embeddings and enables efficient similarity search. Examples: Pinecone, Weaviate, FAISS.

Chunking

Splits large documents into smaller pieces for more efficient processing and retrieval.

Retriever

Fetches the most relevant chunks for the user's question using vector similarity.

Best Practices

Optimized Chunk Size
Use chunks of 200-500 tokens. Too small loses context, too large can dilute relevance.
Chunk Overlap
Maintain 10-20% overlap to preserve context at chunk boundaries.
Rich Metadata
Include source, date, section, and other metadata to filter and rank results.
Reranking
Use a reranking model to refine vector search results.

Building a RAG System

Key architectural decisions and engineering considerations for production-grade RAG pipelines.

Data Preparation

Clean, normalize, and deduplicate your source documents. Remove boilerplate, headers/footers, and irrelevant content. Convert all formats (PDF, DOCX, HTML) to a uniform representation before chunking. Data quality directly determines RAG quality.

Embedding Model Selection

Choose between general-purpose (OpenAI text-embedding-3, Cohere embed-v3) or domain-specific models. Consider dimensionality, multilingual support, max token length, and cost. Benchmark with your actual data using metrics like recall@k before committing.

Vector Database Choice

Evaluate Pinecone (managed), Weaviate (hybrid search), Qdrant (filtering), Chroma (lightweight), pgvector (PostgreSQL). Consider: managed vs self-hosted, metadata filtering, multi-tenancy, scaling, and cost per million vectors.

Chunking Strategy

Match your chunking to your data type: semantic chunking for mixed-topic documents, recursive character splitting for general text, section-aware for structured docs. Track chunk sizes vs retrieval precision to find the optimum.

Retrieval Strategy

Design a multi-stage retrieval pipeline: initial broad retrieval (top-k=20-50), reranking with a cross-encoder, and final selection (top-k=3-5). Consider hybrid retrieval combining dense vectors with sparse (BM25) for better coverage.

Prompt Engineering for RAG

Structure prompts with clear roles: system instruction, retrieved context block with source markers, and user query. Instruct the model to cite sources, say "I don't know" when context is insufficient, and distinguish between retrieved facts and general knowledge.

Indexing Pipeline Design

Build an incremental indexing pipeline with change detection (hash-based), versioning, and rollback. Support both bulk ingestion and real-time updates. Include validation checks: embedding dimension consistency, duplicate detection, and metadata schema enforcement.

Document Preprocessing

Extract text from complex formats using specialized parsers (Unstructured, LlamaParse, Apache Tika). Handle tables, images (OCR/multimodal), and nested structures. Preserve document hierarchy and cross-references. Enrich with metadata: source, date, author, section path.

Ensuring Accuracy

Techniques and strategies to minimize hallucinations and maximize the reliability of RAG-generated responses.

Grounding Techniques

Force the LLM to base answers strictly on retrieved content. Use system prompts that explicitly constrain the model: "Answer ONLY based on the provided context. If the context does not contain sufficient information, say so."

Citation & Attribution

Require the model to include inline citations referencing specific source documents and chunks. Use numbered references [1], [2] mapped to source metadata. This enables verification and builds user trust in the system's outputs.

Confidence Scoring

Implement confidence signals at multiple levels: retrieval similarity scores, LLM self-assessed confidence, and post-hoc consistency checks. Flag low-confidence answers for human review. Use calibrated thresholds based on your domain requirements.

Hallucination Detection

Deploy post-generation checks: verify each claim against retrieved context using NLI (Natural Language Inference) models. Tools like Vectara HHEM, Lynx, or custom classifiers can detect unsupported statements. Reject or rewrite answers that contain ungrounded claims.

Context Window Management

Optimize how retrieved chunks fill the LLM's context window. Order chunks by relevance (most relevant first or "lost in the middle" mitigation). Avoid exceeding the model's effective context length. Consider summarizing lengthy contexts to preserve critical information.

Query Rewriting

Transform user queries to improve retrieval: expand abbreviations, resolve ambiguity, decompose complex questions into sub-queries, and generate hypothetical document embeddings (HyDE). Use the LLM itself to reformulate queries before retrieval.

Hybrid Search

Combine dense vector search (semantic similarity) with sparse search (BM25/TF-IDF keyword matching). This captures both semantic meaning and exact keyword matches. Use reciprocal rank fusion (RRF) or learned weights to merge results from both approaches.

Cross-Encoder Reranking

After initial retrieval, apply a cross-encoder model (e.g., Cohere Rerank, BGE Reranker, ColBERT) to re-score query-document pairs. Cross-encoders are more accurate than bi-encoders for relevance but too slow for initial retrieval, making them ideal for reranking a short candidate list.

Source Verification

Validate that retrieved documents are authoritative and current. Implement freshness checks against document timestamps, cross-reference claims across multiple retrieved chunks, and flag answers derived from a single source. Maintain an audit trail from answer back to source document.

Quality Validation & Evaluation

Systematic approaches to measuring and improving RAG system performance with quantitative and qualitative methods.

The RAG Triad

The three pillars of RAG quality evaluation. Every RAG response should be assessed against these three dimensions.

Context Relevance

Are the retrieved documents actually relevant to the user's query? Measures retrieval precision. Low context relevance means the retriever is pulling irrelevant chunks, adding noise to the LLM's input. Improve with better embeddings, query rewriting, or reranking.

Groundedness

Is the answer actually supported by the retrieved context? Detects hallucinations where the LLM generates information not present in the source documents. Use NLI-based checks to verify each statement in the response is entailed by the context.

Answer Relevance

Does the answer actually address the user's original question? A response can be grounded and use relevant context but still miss the point. Measure by generating potential questions from the answer and comparing them with the original query via semantic similarity.

Evaluation Frameworks

RAGAS

Open-source framework providing automated metrics for faithfulness, answer relevance, context precision, and context recall. Supports LLM-based evaluation and integrates with LangChain and LlamaIndex. The de facto standard for RAG evaluation.

DeepEval

Testing framework for LLM applications with 14+ metrics including hallucination, toxicity, and bias detection. Features a Pytest-like interface, making it natural for engineering teams to integrate RAG testing into CI/CD pipelines.

TruLens

Instrumentation and evaluation framework that traces the full RAG pipeline. Provides feedback functions for the RAG Triad, tracks experiments across iterations, and offers a dashboard for visualizing quality trends over time.

Human Evaluation

Create structured rubrics for human annotators to evaluate answer quality, correctness, and completeness. Use inter-annotator agreement (Cohen's Kappa) to ensure consistency. Combine with automated metrics for a complete evaluation strategy. Essential for high-stakes domains like healthcare and legal.

A/B Testing

Deploy multiple RAG configurations simultaneously and compare user satisfaction, task completion rates, and answer quality metrics in production. Use statistical significance tests to determine which configuration performs better. Test one variable at a time: embedding model, chunk size, retrieval strategy, or prompt template.

Golden Dataset Creation

Build a curated test set of question-answer-context triples that represent your domain's query distribution. Include edge cases, multi-hop questions, and "unanswerable" queries. Version and expand this dataset over time. Use it as the ground truth for regression testing every pipeline change.

Key Automated Metrics

Faithfulness

Measures whether every claim in the answer can be traced back to the retrieved context. Score 0-1.

Answer Relevance

Measures how well the answer addresses the original question. Penalizes incomplete or tangential responses.

Context Precision

Of all retrieved documents, what proportion is actually relevant? High precision = less noise in context.

Context Recall

Of all relevant documents in the corpus, what proportion was retrieved? High recall = fewer missed answers.

Risks & Common Pitfalls

Understand the failure modes and anti-patterns that can undermine your RAG system in production.

Critical Risks

Data Quality Issues
Garbage in, garbage out. Poorly formatted, inconsistent, or inaccurate source documents will poison every downstream component. OCR errors, broken tables, and encoding issues are silent killers of RAG quality.
Context Poisoning
Malicious or adversarial content injected into the knowledge base can manipulate RAG outputs. An attacker could insert documents containing instructions that override the system prompt, leading to prompt injection via retrieval.
Hallucination Despite Retrieval
Even with relevant context, LLMs can still hallucinate by extrapolating, merging information from different sources incorrectly, or generating plausible but unsupported conclusions. Having retrieval does not guarantee factual accuracy.
Privacy & Data Leakage
RAG systems can inadvertently expose sensitive information from the knowledge base. Without proper access controls, user A could receive answers containing user B's confidential data. Implement document-level permissions and query-time access filtering.
Prompt Injection via Retrieval
Retrieved documents may contain adversarial text that hijacks the LLM's behavior. For example, a document might contain "Ignore all previous instructions and..." which, when injected into the prompt, overrides your system instructions. Sanitize retrieved content and use output guards.

Operational Pitfalls

Stale Data

Knowledge bases that are not regularly updated lead to incorrect or outdated answers. Implement automated sync pipelines with freshness monitoring and alerts when documents exceed their expected refresh interval.

Over-Reliance on Retrieval

Not every query requires retrieval. Forcing retrieval for general knowledge questions adds latency and may introduce irrelevant context that hurts answer quality. Implement query routing to classify whether retrieval is needed.

Embedding Drift

When you update or change your embedding model, existing vectors become incompatible with new ones. This requires full re-indexing of your knowledge base. Plan for embedding model migration and maintain version compatibility tracking.

Scalability Bottlenecks

Vector search latency grows with corpus size. Embedding computation becomes a bottleneck during ingestion. LLM context window costs increase with more retrieved chunks. Plan capacity for 10x your current data volume and query rate.

Bias in Training Data

Your RAG system inherits biases present in the knowledge base. If source documents over-represent certain perspectives, the system's answers will reflect that skew. Audit your corpus for representation, perform fairness evaluations, and document known biases.

Cost Management

RAG costs compound across three axes: embedding computation (per token), vector storage (per vector/month), and LLM inference (per token in context). Monitor cost per query, optimize chunk sizes, cache frequent queries, and consider smaller models for reranking steps.

Testing Strategies

Comprehensive testing approaches to validate RAG pipeline quality at every stage, from unit tests to production monitoring.

Testing Types

Unit Testing

Test each component in isolation: verify chunking produces expected splits, embeddings have correct dimensions, retriever returns relevant documents for known queries. Mock external services (LLM, vector DB) and assert deterministic behaviors.

Integration Testing

Test the full pipeline end-to-end: query in, answer out. Verify the complete flow from query embedding through retrieval, context assembly, and generation. Compare outputs against golden answers using semantic similarity and exact-match metrics.

Regression Testing

Maintain a golden test set and run it against every pipeline change: new embedding models, updated prompts, changed chunk sizes. Track metric trends over time. A change that improves one metric should not regress another. Automate in CI/CD.

Stress & Load Testing

Measure system behavior under peak load: concurrent queries, large document ingestion batches, and sustained high throughput. Identify bottlenecks in embedding computation, vector search, and LLM inference. Set SLAs for p50, p95, and p99 latencies.

Adversarial Testing

Probe the system with malicious inputs: prompt injections embedded in queries, adversarial documents in the knowledge base, intentionally misleading questions, and queries designed to extract sensitive information. Build a red-team test suite.

Edge Case Testing

Test boundary conditions: empty queries, extremely long queries, queries in unsupported languages, questions with no relevant context in the knowledge base, multi-hop reasoning requirements, and contradictory information across sources.

Production Monitoring

Track key signals in real-time: retrieval latency (p50/p95/p99), retrieval relevance scores distribution, LLM token usage and cost per query, user feedback (thumbs up/down), error rates, and cache hit ratios. Set up alerts for quality degradation. Use tools like LangSmith, Arize Phoenix, or custom Prometheus/Grafana dashboards.

Evaluation Metrics Reference

MRR (Mean Reciprocal Rank)

Average of 1/rank of the first relevant result. Range: 0-1. Higher = relevant docs appear earlier in results.

NDCG (Normalized Discounted Cumulative Gain)

Measures ranking quality considering the position and graded relevance of all results. Range: 0-1. The gold standard for search ranking evaluation.

F1 Score

Harmonic mean of precision and recall for answer token overlap. Balances the trade-off between finding all relevant information and avoiding irrelevant content.

Hit Rate (Recall@k)

Proportion of queries where at least one relevant document appears in the top-k results. The most intuitive retrieval metric. Target: >95% at k=10 for production systems.