Design a Production RAG System

A model alone stores knowledge in parameters that are expensive to update and difficult to cite. RAG keeps source knowledge in external indexes and retrieves it at query time. The LLM is then used as a reasoning and synthesis layer, not as the only source of truth.

This separation is valuable when knowledge changes often, when the business needs citations, or when answers must respect tenant permissions. A common target is to retrieve 20 to 100 candidate chunks, rerank down to 5 to 12 evidence chunks, and fit those into a 4K to 32K token context budget with room for instructions and the final answer.

The unit you embed determines what the retriever can find. Small chunks such as 200 to 400 tokens improve pinpoint recall but may lose context. Large chunks such as 1000 to 1600 tokens preserve narrative but dilute the vector and waste prompt budget. A practical default for knowledge docs is 600 to 900 tokens with 80 to 150 token overlap.

Good systems store both chunk text and rich metadata: document id, version, section path, title, author, timestamp, tenant id, access labels, language, source URL, and token offsets. Metadata enables filtering, citations, freshness, ACL enforcement, and targeted reindexing.

Dense vector search captures semantic similarity, paraphrases, and natural-language intent. It can match refund eligibility to cancellation policy even when exact words differ. BM25 captures exact terms, product names, error codes, SKUs, legal phrases, and rare identifiers that embeddings may blur.

Hybrid retrieval runs both and combines them with reciprocal rank fusion, weighted normalized scores, or learned fusion. In many enterprise RAG systems, hybrid retrieval improves recall@20 by 5 to 20 percentage points compared with dense-only search, especially when queries contain acronyms, version numbers, or code symbols.

Approximate nearest neighbor search is optimized for recall and speed, not final ordering. A cross-encoder reranker reads the full query and candidate chunk together, then predicts relevance more accurately than independent embeddings. It is usually applied to the top 40 to 200 candidates, not the full corpus.

The tradeoff is latency and cost. A small reranker can score 100 pairs in 30 to 120 ms on a GPU batch, while a larger model may take 150 to 500 ms. Production designs cap candidate count, batch across concurrent requests, and skip reranking for high-confidence cache hits or simple navigational queries.

A grounded answer is not just an answer with retrieved text nearby. The prompt must tell the model to use only supplied evidence, cite source identifiers, and say when the evidence is insufficient. The context builder should preserve source boundaries so citations map back to exact chunks, pages, or URL anchors.

Citation quality is a product feature and an evaluation target. Teams often track citation precision, citation recall, unsupported claim rate, and answer faithfulness using a mix of human labels, LLM judges, and adversarial test sets.

Enterprise RAG fails if stale or unauthorized content is retrieved. Every query should carry tenant, user, role, region, and policy attributes into retrieval filters. Every chunk should carry document version, deletion state, timestamp, and ACL metadata.

Freshness service-level objectives vary by domain. Product docs may tolerate 15 minute lag. Incident runbooks may require under 2 minutes. Legal records may require immediate delete propagation. The architecture should support incremental updates and tombstones rather than nightly full reindexing only.

A strong default is hierarchical chunking. Preserve document structure, split by heading and paragraph, then enforce token limits. Use 600 to 900 tokens for long-form knowledge articles, 200 to 400 tokens for FAQs, and 1000 to 1600 tokens for legal or policy content where definitions depend on nearby context. Keep overlap around 10 to 20 percent, such as 80 to 150 tokens.

Chunk metadata is as important as chunk text. Store source_doc_id, source_version, title, section_path, page number, URL anchor, created_at, updated_at, language, tenant_id, ACL labels, content_hash, and embedding_model. This enables filtered retrieval, citation rendering, incremental reindexing, and delete propagation.

Do not blindly chunk every file the same way. Tables may need row groups plus header repetition. Code needs symbol-aware chunks. Slide decks need title plus speaker notes. PDFs need page coordinates if the product shows source previews.

Embedding dimension affects quality, memory, and latency. A 768 dimensional float32 vector costs about 3 KB raw. A 1536 dimensional vector costs about 6 KB raw. A 3072 dimensional vector costs about 12 KB raw. With HNSW graph overhead, metadata, replicas, and deleted tombstones, production memory can be 2x to 5x raw vector size.

For 100M chunks at 1536 dimensions with float32, raw vectors alone are about 614 GB. With HNSW overhead and replicas, plan multiple TB. Quantization to int8 or product quantization can reduce memory by 4x to 16x but may lower recall. If the corpus is under 10M chunks, HNSW is often simple and fast. At hundreds of millions to billions, consider IVF, disk-backed ANN, sharding, and tiered hot versus cold indexes.

Index targets should be explicit: recall@50 above 0.90 on labeled queries, vector search p95 under 80 ms in-region, and ingestion lag under the product freshness SLO.

Dense search and BM25 should both run under the same authorization filters. If filtering is applied after retrieval, unauthorized chunks may affect ranking or leak through logs. Prefer pre-filtering inside the vector DB and keyword engine when selectivity is reasonable. For very selective filters, maintain tenant shards or filtered indexes for large tenants.

Reciprocal rank fusion is a robust baseline because it uses ranks rather than raw scores. A common formula gives each result a score of 1 divided by k plus rank, with k around 60. It works well when vector scores and BM25 scores have different scales. Weighted normalized fusion can perform better after calibration, for example 0.65 dense and 0.35 BM25 for natural language support queries, or the reverse for code and SKU queries.

The retriever should also support query rewriting. For conversational questions, rewrite the latest turn into a standalone query. For acronyms, expand known aliases. For multilingual corpora, detect language and either search same-language indexes or translate the query while preserving named entities.

A cross-encoder jointly attends to the query and candidate text, so it sees exact phrase matches, negations, and context that a bi-encoder may miss. This improves precision at the final context size, especially when the top vector results are semantically close but not answer-bearing.

The reranker is expensive because each query and candidate pair is a separate model input. Keep the candidate set bounded, such as 40, 80, or 120 chunks. Batch pairs across a single query and across concurrent queries. Use a smaller model for p95 latency and reserve larger rerankers for offline labeling or high-value enterprise tenants.

Typical latency budgets are 20 to 80 ms for vector and BM25 retrieval, 50 to 150 ms for reranking 80 pairs on GPU, 20 to 60 ms for context assembly, and 800 to 2500 ms for LLM generation depending on answer length.

Context assembly is where many RAG systems become answer engines. The builder should remove near-duplicates, cap chunks per document, diversify sources, preserve headings, and attach stable citation ids. It should not simply concatenate top chunks. If two chunks overlap heavily, keep the higher ranked one and merge adjacent context only when it improves answerability.

A useful context record includes citation_id, source_title, source_url, section_path, timestamp, and chunk_text. The prompt can then ask the model to cite citation_id values only. After generation, validate that every citation id exists and optionally verify that cited chunks support nearby claims using a lightweight entailment model or LLM judge.

Token budgeting matters. With a 16K model context, you may reserve 2K for system and developer instructions, 2K for conversation history, 8K to 10K for evidence, and 2K to 4K for the response. For 128K context models, avoid stuffing everything by default because cost and attention dilution can hurt quality.

Offline retrieval evaluation needs labeled query and relevant document pairs. Track recall@5, recall@10, recall@20, MRR, nDCG, and coverage by source, tenant, language, and query type. A healthy flagship system might target recall@20 above 0.90 for curated enterprise QA and recall@10 above 0.80 for messy long-tail support queries.

Generation evaluation measures faithfulness, answer completeness, citation accuracy, refusal correctness, and harmful or policy-violating output. Use human review for golden sets, LLM judges for scale, and adversarial cases for prompt injection, stale docs, conflicting docs, and ambiguous questions.

Online metrics should connect user behavior to retrieval traces: cache hit rate, no-answer rate, citation click rate, escalation rate, feedback score, latency p50 and p95, tokens per answer, cost per answer, and answer regeneration rate.

Semantic caching can reduce cost and latency when many users ask similar questions. Cache exact normalized queries first, then semantic matches using query embeddings and a similarity threshold such as cosine 0.92 to 0.97. Store the original query, answer, citations, retrieval ids, prompt version, model version, tenant, policy hash, and index watermark.

The hard part is invalidation. An answer about pricing, incident status, or security policy must expire quickly or be validated against source versions. Cache TTLs may range from 5 minutes for fast-changing docs to 7 days for stable conceptual docs. If any cited document has a newer version than the cached answer watermark, force retrieval and regeneration.

Personalized answers should be cached only within the same tenant and policy boundary. When in doubt, cache retrieved candidate ids or context plans rather than final answers.

RAG cost is a sum of ingestion and serving. Ingestion cost includes parsing, OCR, embedding, vector storage, keyword indexing, and reindexing. Serving cost includes query embedding, vector and BM25 search, reranking, LLM input tokens, LLM output tokens, logging, and evaluation.

Example serving cost for one answer: one query embedding at $0.02 to $0.13 per 1M tokens, reranking 80 pairs at perhaps $0.05 to $0.50 per 1000 queries depending on hosting, 7000 LLM input tokens at $2.50 per 1M tokens, and 700 output tokens at $10.00 per 1M tokens. That is roughly a few cents for a high-quality answer on premium models, and much less on smaller models.

Latency is usually dominated by LLM output tokens. Retrieval and reranking should be optimized enough that generation remains the main visible delay. Streaming the answer can hide some latency, but citations should still be stable by the time the final response is rendered.

Trace every answer with request_id, user policy hash, query rewrite, retrieval filters, vector top-k, BM25 top-k, fusion scores, reranker scores, selected context ids, prompt version, model version, token counts, latency breakdown, cache decision, citations, and feedback. Without this trace, debugging a hallucination becomes guesswork.

Dashboards should show ingestion lag, parse failures, index write failures, vector search p95, keyword search p95, reranker p95, LLM p95, cache hit rate, recall on canary sets, faithfulness score, unsupported claim rate, and cost per 1000 answers.

Offline ingestion should be retryable and idempotent. Use document version plus chunk hash as natural idempotency keys. If embedding APIs fail, retry with exponential backoff and keep the document in a pending state rather than serving partial indexes silently.

Online requests need stricter timeouts. If vector search exceeds 150 ms or reranking exceeds 250 ms, return degraded retrieval or a polite retry depending on product requirements. Apply backpressure before queues grow enough to make answers stale.

Treat embeddings, chunkers, fusion logic, rerankers, prompts, and models as versioned artifacts. A new embedding model requires either dual-writing new vectors or building a shadow index before cutover. Compare recall@k and answer quality on golden queries before moving traffic.

Use canary deployments by tenant or percentage of traffic. Keep old indexes available until rollback risk is low. Log index_version and embedding_model on every retrieved chunk so evaluation can compare versions.

Source deletes and ACL changes must propagate quickly. Store tombstones so old chunks are filtered immediately even before compaction. For sensitive domains, enforce ACLs at retrieval time and again before context assembly. Do not rely on prompt instructions to hide unauthorized evidence.

Freshness SLOs should be explicit. A support help center might require 15 minute p95 ingestion lag. Security incident docs may require 2 minute lag. Legal deletion requests may require immediate exclusion from online retrieval and later physical deletion from indexes.

RAG retrieves untrusted text. A malicious document can say ignore previous instructions or reveal secrets. The prompt should separate instructions from evidence and explicitly state that retrieved content is data, not instructions. The context builder should strip active HTML, scripts, hidden text, and suspicious boilerplate.

For high-risk products, add detectors for prompt injection patterns, secrets, PII, and policy violations. Log and quarantine suspicious documents during ingestion.

Use cheaper models for query rewriting, classification, and simple answers. Route complex synthesis to larger models only when necessary. Cap evidence tokens, output tokens, and reranker candidates per tenant tier. Cache frequent answers and retrieve-only snippets when generation is unnecessary.

Track cost per tenant and per feature. Good dashboards show embedding spend, LLM input spend, LLM output spend, reranker GPU utilization, and wasted cost from no-answer or low-feedback sessions.

If the reranker is down, fall back to fused retrieval scores. If the LLM provider is unavailable, return top cited snippets with a degraded message. If a source connector is lagging, expose freshness warnings for affected sources. If retrieval confidence is low, ask a clarifying question or abstain instead of guessing.

The important interview point is to define fallback by risk. Customer support may tolerate a slower answer. Compliance may prefer no answer. Internal search may return documents without synthesis.

This simplified Python example shows deterministic chunk ids and metadata propagation. Production chunkers should split on headings and semantic boundaries before falling back to token limits, but the mechanics of overlap and stable ids are the same.

This example merges dense and BM25 result lists without assuming their scores are comparable. It is a strong baseline for interviews because it is simple, stable, and works before you have enough data to train a learned fusion model.

The context builder should keep evidence structured until the prompt is assembled. Stable citation ids let the model cite evidence without inventing source names, and they let the application validate the final answer.