Design a RAG Pipeline (LLD)
Loaders, chunkers, embedders, a vector store, retrievers, and a generation step composed as a pipeline.
Problem Statement
Design the low-level object model for a Retrieval Augmented Generation pipeline. The system ingests documents, chunks them, creates embeddings, stores vectors, retrieves the most relevant chunks for a user question, assembles a grounded prompt, and sends that prompt to a generator.
The interview focus is not model quality or distributed indexing. It is the class design: clean stage boundaries, pluggable chunking and embedding strategies, swappable vector-store implementations, and a chain that can be extended without rewriting the pipeline.
Business context
RAG systems are common in enterprise search, support assistants, compliance copilots, and internal knowledge bots. Teams rarely use one permanent embedding vendor or vector database; they evaluate local models, OpenAI or Azure embeddings, Pinecone, pgvector, Elasticsearch, and in-memory stores during development.
A strong LLD answer keeps the retrieval path independent from those vendor decisions. The core pipeline should say what happens at each stage, while adapters and factories decide how a concrete provider or store is used.
Functional Requirements
Accept one or more source documents with id, title, text, and metadata.
Split each document into ordered chunks using a pluggable chunking strategy.
Embed chunks through an embedding provider interface that can adapt different vendor clients.
Persist embedded chunks in a vector store interface with swappable backends.
Retrieve the top K chunks for a user question using query embedding plus vector similarity.
Assemble a grounded prompt from the question and retrieved chunks.
Generate an answer through an injectable generator step.
Wire ingestion, chunking, embedding, indexing, retrieval, prompt assembly, and generation as a chain of stages.
Non-Functional Requirements
Provider portability
Embedding and vector storage must be replaceable without changing the pipeline stages or retriever.
Deterministic boundaries
Every stage consumes and produces clear domain objects so failures are easy to isolate and retry.
Low coupling
Vendor SDKs stay behind adapters. The domain never imports a Pinecone, pgvector, OpenAI, or Azure client type.
Testability
Unit tests should run with a local hash embedding provider and in-memory vector store, while production uses real backends.
Incremental extensibility
Adding reranking, filtering, caching, or moderation should mean adding a stage or strategy, not editing every existing class.
Requirement Clarification
QAre we designing offline ingestion or online question answering?
Model both paths in one pipeline for the LLD exercise. Documents are indexed first, then the same request retrieves chunks and generates an answer.
QDo embeddings come from one fixed model?
No. Treat embeddings as a provider interface. The implementation includes a local hashing provider for demos and an adapter method for vendor clients.
QWhich vector database should we assume?
Do not hard-code one. Define a VectorStore interface and ship an in-memory implementation; production can plug in pgvector, Pinecone, Elasticsearch, or another backend.
QDo we need semantic reranking, citations, or streaming tokens?
Not in the base design. Keep retrieval top K and prompt assembly simple, then call out reranking, citation formatting, and streaming as extensions.
QShould chunks be mutable once indexed?
No. Model Document and Chunk as immutable values. Attaching an embedding returns a new chunk so indexing cannot observe partially mutated state.
UML Class Diagram
Sequence Diagram
Entity Identification
Document
Immutable source text with identity and metadata. It represents the raw unit of ingestion before chunking.
Chunk
Immutable fragment derived from a document. It carries document lineage, order, text, and an optional embedding vector.
ChunkingStrategy
Strategy interface for splitting documents. Fixed-size, sentence-aware, markdown-aware, or semantic chunkers can share the same contract.
EmbeddingProvider
Provider abstraction for turning text into vectors. The adapter factory wraps vendor clients behind the same embed method.
VectorStore
Storage abstraction for embedded chunks. It supports upsert and similarity search while hiding the chosen backend.
Retriever
Embeds the user query, asks the vector store for top K chunks, and returns ranked search results.
RagPipeline
Facade over a chain of stages: validate, chunk, embed, index, retrieve, assemble prompt, and generate answer.
StageContext
Internal request state passed through the chain. It prevents long method signatures and keeps intermediate data scoped to one run.
Design Patterns Used
ChunkingStrategy, EmbeddingProvider, prompt assembly, and generation are behavior seams. The pipeline calls interfaces while concrete choices are injected.
RagPipeline links stages so ingestion, chunking, embedding, indexing, retrieval, prompt assembly, and generation can be extended or reordered without one large method.
EmbeddingProvider.fromClient adapts any vendor-specific embedding client into the domain interface, keeping SDK classes out of the core model.
Static factory methods such as ChunkingStrategy.fixedSize, EmbeddingProvider.localHashing, EmbeddingProvider.fromClient, and VectorStore.inMemory create concrete implementations behind interfaces.
Step-by-Step Design
1Start with immutable ingestion values
Make Document and Chunk immutable. A chunk with an embedding is a new value, so callers never observe a half-indexed chunk.
Chunk embedded = chunk.withEmbedding(embeddingProvider.embed(chunk.getText())); vectorStore.upsert(java.util.Collections.singletonList(embedded));
2Hide chunking policy behind a strategy
Different corpora need different splitting rules: fixed-size windows, sentence boundaries, markdown sections, or semantic windows. The pipeline only depends on chunk(document).
ChunkingStrategy chunker = ChunkingStrategy.fixedSize(500, 80); List<Chunk> chunks = chunker.chunk(document);
3Adapt embedding vendors at the boundary
Do not let SDK clients leak into the pipeline. Wrap them once and expose a stable EmbeddingProvider interface to the rest of the design.
EmbeddingProvider provider = EmbeddingProvider.fromClient(text -> externalClient.embed(text)); double[] vector = provider.embed("refund policy");4Make vector storage a backend-neutral contract
VectorStore exposes upsert and similaritySearch only. In-memory storage supports tests; production stores can implement the same methods with indexes and filters.
5Keep retrieval small and injectable
Retriever embeds the question, asks the store for nearest chunks, and returns ranked results. It does not know how chunks were generated or which database answered the query.
Retriever retriever = new Retriever(provider, vectorStore, 3); List<VectorStore.SearchResult> hits = retriever.retrieve("How do we deploy safely?");6Wire the flow as a chain of stages
The chain keeps orchestration readable and open for additions such as reranking, query rewriting, access filtering, caching, or moderation.
RagPipeline pipeline = new RagPipeline.Builder() .chunkingStrategy(ChunkingStrategy.fixedSize(500, 80)) .embeddingProvider(EmbeddingProvider.localHashing(64)) .vectorStore(VectorStore.inMemory()) .topK(3) .build();7Assemble prompts after retrieval, not before
Prompt assembly should consume ranked chunks and the original question. Keeping it as a strategy makes it easy to add citations, token budgets, safety text, or model-specific formatting.
Complete Java Implementation
Explanation of Every Class
Document
Immutable ingestion record. It validates identity, title, text, and copies metadata so downstream stages cannot mutate the source while a run is in progress.
Chunk
Immutable indexed fragment. withEmbedding returns a new chunk carrying a defensive copy of the vector, preserving clean stage handoff.
ChunkingStrategy
Strategy interface for document splitting. The included fixedSize factory creates an overlapping window chunker; sentence-aware or markdown-aware chunkers can implement the same method.
EmbeddingProvider
Embedding abstraction and adapter seam. localHashing is deterministic for tests, while fromClient wraps a real provider client behind the same embed contract.
VectorStore
Backend-neutral vector index. inMemory demonstrates upsert and cosine similarity; production backends keep the same interface but delegate to specialized indexes.
Retriever
Small query-time component. It embeds the question, asks the store for top K nearest chunks, and returns ranked search results without knowing backend details.
RagPipeline
Builds and executes the chain of responsibility. The builder injects strategies and stores, then stages validate, chunk, embed, index, retrieve, assemble, and generate.
Main
Demo composition root. It constructs documents, picks fixed-size chunking, local embeddings, an in-memory vector store, and runs the pipeline end to end.
Dry Run
Sample input
Documents: Deployment Handbook and Incident Playbook. Query: How should we deploy after an incident? Use fixed-size chunking, local hashing embeddings, in-memory vector store, and topK = 2.
| Step | Stage | Context change | Pluggable seam | Result |
|---|---|---|---|---|
| 1 | Ingestion | Validate two immutable documents | Document loader could change | Context has 2 documents |
| 2 | Chunking | Split documents into overlapping chunks | ChunkingStrategy | Context has ordered chunks |
| 3 | Embedding and indexing | Attach vectors and upsert chunks | EmbeddingProvider plus VectorStore | Store contains embedded chunks |
| 4 | Retrieval | Embed query and run similarity search | Retriever topK and VectorStore backend | Top 2 incident and deployment chunks returned |
| 5 | Prompt and generation | Build grounded prompt and call generator | PromptAssembler plus Generator | Answer cites release freeze, canary, rollback, and approval guidance |
The important observation is that the chain order stays stable while the embedding provider and vector store can be swapped at construction time.
Complexity Analysis
| Operation | Time | Space | Note |
|---|---|---|---|
| ingest and chunk N characters | O(N) | O(C) | C is the number of chunks produced by the chosen chunking strategy. |
| embed and index C chunks | O(C × E + storeUpsert) | O(C × D) | E is embedding-provider cost and D is embedding dimension. |
| in-memory similarity search | O(C × D + C log C) | O(C) | The demo store scores every chunk and sorts. Approximate indexes improve this in production. |
| prompt assembly | O(K × L) | O(K × L) | K retrieved chunks and L average chunk length are copied into the prompt. |
The LLD design intentionally hides backend complexity. A production vector store may use HNSW, IVF, disk-backed indexes, or metadata filters, but the retriever still calls similaritySearch(queryEmbedding, topK).
Extensibility
Swap embedding model
Replace EmbeddingProvider.localHashing with EmbeddingProvider.fromClient wrapping Azure OpenAI, OpenAI, Cohere, local ONNX, or another model.
Swap vector backend
Implement VectorStore for pgvector, Pinecone, Elasticsearch, Redis, or a sharded service. Retriever and RagPipeline do not change.
Add reranking
Insert a RerankingStage after retrieval or decorate Retriever. It can call a cross-encoder and keep the same prompt assembly contract.
Add access control
Add a metadata filter before similarity search or a stage that removes chunks the caller cannot see before prompt assembly.
Add citations
Extend Chunk metadata and prompt assembly to include source title, section, and offsets. No embedding or storage contract needs to change.
Alternative Designs
One service class with private helper methods
Put ingest, chunk, embed, retrieve, prompt, and generation methods inside one RagService.
Tradeoffs
Fast to code in an interview, but policy decisions become private methods and stage insertion requires editing the central class.
Event-driven ingestion and query services
Split document ingestion into an asynchronous worker and keep online query answering in a separate service.
Tradeoffs
Better production scalability, retries, and isolation, but more infrastructure than an LLD interview usually needs.
Repository-style vector store only
Treat vector storage as a repository and keep retrieval logic in the application service.
Tradeoffs
Simple for CRUD-like systems, but it mixes query embedding, nearest-neighbor policy, and prompt needs unless Retriever remains a separate object.
Common Mistakes
- ×
Hard-coding one embedding provider directly inside the pipeline.
- ×
Letting vector database SDK types leak into Retriever or RagPipeline.
- ×
Making chunks mutable and attaching embeddings in place across multiple stages.
- ×
Writing one giant answer method instead of isolated stages with clear responsibilities.
- ×
Treating prompt assembly as string concatenation hidden inside the generator, which makes citations and token limits difficult.
- ×
Ignoring the duplicate-indexing problem when the same document is ingested repeatedly.
- ×
Returning raw chunks from storage without score or source information for prompt assembly.
Follow-up Interview Questions
QHow would you add Pinecone or pgvector?
Create a new VectorStore implementation. It maps Chunk plus embedding into backend records and maps search results back into VectorStore.SearchResult.
QHow do you support multiple embedding models at once?
Keep model identity in index configuration or chunk metadata, and route both indexing and query embedding through a named EmbeddingProvider. Never compare vectors from different dimensions or models.
QWhere should token budgeting live?
In prompt assembly or a dedicated ContextPackingStage after retrieval. It can trim chunks by score, recency, or source priority before generation.
QHow would you make ingestion asynchronous?
Split the chain after document validation: publish chunking and embedding jobs to a queue, then keep online query answering as retrieval plus prompt plus generation.
QHow do you prevent unauthorized chunks from entering the answer?
Carry ACL metadata on documents and chunks, filter before or during vector search, and verify again before prompt assembly as a defense-in-depth stage.
Production Considerations
Idempotent indexing
Use stable chunk ids based on document id, version, and chunk index. Upsert should replace old chunks for the same version and delete stale chunks for superseded versions.
Observability
Track chunk counts, embedding latency, store upsert latency, retrieval hit scores, prompt token count, generator latency, and grounded-answer feedback.
Failure handling
Retry transient embedding and vector-store failures with backoff. Persist ingestion state so a failed document can resume without duplicating chunks.
Security and privacy
Encrypt stored vectors and source text, avoid sending restricted data to unauthorized providers, and enforce metadata filters before prompt assembly.
Evaluation
Maintain golden queries with expected source chunks. Measure retrieval recall, answer grounding, latency, and cost after provider or store swaps.
What Interviewers Look For
Did you separate ingestion, chunking, embedding, indexing, retrieval, prompt assembly, and generation responsibilities?
Can the candidate swap embedding providers and vector stores without editing Retriever or RagPipeline?
Is the Chain of Responsibility useful, or did it become ceremony around a simple method?
Are immutable Document and Chunk values used to avoid hidden state changes between stages?
Did the answer mention production concerns such as idempotent indexing, metadata filters, latency, evaluation, and provider failures?
Quiz
0/5 answered
1.Why is **EmbeddingProvider** an interface?
2.Which component should know how to call Pinecone or pgvector?
3.What does Chain of Responsibility buy in this design?
4.Where should prompt token budgeting most naturally live?
5.Why should chunks be immutable?
Practice Variants
Add a reranking stage
AdvancedInsert a stage after retrieval that reranks top 20 vector hits down to top 5 with a cross-encoder strategy.
Add metadata-aware retrieval
ExpertExtend VectorStore.similaritySearch to accept filters for tenant, document type, ACL, and freshness while keeping Retriever small.
Support streaming generation
ExpertChange Generator to emit tokens through an observer or callback while preserving the same retrieval and prompt assembly stages.
Flashcards
Cheat Sheet
Entities: Document, Chunk, ChunkingStrategy, EmbeddingProvider, VectorStore, Retriever, RagPipeline.
Flow: validate documents → chunk → embed chunks → upsert vectors → embed query → similarity search → assemble prompt → generate answer.
Patterns: Strategy for chunking and providers; Adapter for vendor embedding clients; Factory Method for default implementations; Chain of Responsibility for stages.
Key seams: embedding provider, vector store, retriever top K, prompt assembler, generator, optional reranker.
Invariants: chunks retain document lineage; embeddings match the configured model dimension; vector store only indexes embedded chunks; prompt assembly uses retrieved context.
Production checks: idempotent indexing, ACL filters, provider retries, token budgets, retrieval evaluation, observability, and data privacy.
References
- BookDesigning Data-Intensive Applications — Martin Kleppmann
- PaperRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al.
- DocsOpenAI Embeddings Guide
- DocsPinecone Vector Database Concepts