Design an AI Chat Platform (LLD)
Conversations, messages, model providers, streaming, tools, and context windows modelled in objects.
Problem Statement
Design the low-level object model for a ChatGPT-style AI chat platform. Users create conversations, send messages, receive streamed assistant tokens, switch between model providers such as OpenAI, Anthropic, and a local model, and allow the assistant to call registered tools when the model asks for external data or actions.
The focus is the LLD boundary: conversation/message ownership, a provider interface that hides vendor APIs, streaming delivery through observers, context-window trimming, and a safe dispatch path for function calls.
Business context
AI chat products need to move quickly across providers because cost, quality, latency, and model availability change constantly. A poor design leaks vendor request shapes into the service layer and makes every streaming or tool-calling change risky. A strong LLD keeps the chat domain stable while adapters absorb provider differences and observers make token streaming independent of the UI.
Functional Requirements
Create and maintain conversations with ordered system, user, assistant, and tool messages.
Send a user message and stream the assistant response token by token.
Support multiple model providers behind one ModelProvider abstraction.
Allow provider-specific adapters for OpenAI, Anthropic, and local inference.
Trim conversation history to a configurable context window before each model call.
Register named tools and dispatch function calls requested by a provider.
Broadcast streaming events to request-scoped and conversation-level observers.
Persist the final assistant message after streaming completes successfully.
Non-Functional Requirements
Provider isolation
Chat orchestration must not depend on OpenAI, Anthropic, or local request formats. Provider changes stay inside adapters.
Low-latency streaming
The first token should reach observers as soon as the provider emits it; the service must not wait for the full answer.
Context safety
Every provider call uses a bounded context window so old messages cannot blow past token limits.
Tool-call containment
Tools are registered by name and invoked through a narrow dispatch interface; provider adapters never call arbitrary application methods.
Extensibility
New providers, tools, observers, and memory policies should be additive changes rather than edits to the chat flow.
Requirement Clarification
QAre we designing the full distributed backend?
No. Keep the base design in-memory and class-focused. Persistence, queues, auth, billing, and horizontal scaling are production extensions.
QDo model providers return identical response shapes?
No. That is the reason for adapters. The service only sees ModelProvider.streamChat and ToolCallHandler.call.
QHow accurate does token counting need to be?
For the interview model use a simple estimate on Message. In production, swap in a tokenizer-aware context policy.
QCan tools be called while streaming?
Yes. The provider adapter may request a tool through the handler, receive a string result, and continue emitting tokens.
QDo observers represent WebSockets, SSE, logs, or tests?
All of them. StreamObserver is intentionally transport-neutral; UI delivery is a boundary adapter outside this core model.
UML Class Diagram
Sequence Diagram
Entity Identification
Conversation
Aggregate for ordered chat history. It owns messages and returns a bounded context window for provider calls.
Message
Immutable chat item with a role, content, optional tool name, timestamp, and rough token estimate.
ModelProvider
Strategy interface for model execution. It hides provider choice behind streamChat and exposes a factory method for demos and tests.
OpenAiProvider
Adapter from the platform model interface to an OpenAI-shaped client. It translates context and streams normalized tokens.
AnthropicProvider
Adapter from the platform model interface to an Anthropic-shaped client, including its own prompt composition.
LocalProvider
In-process provider strategy used for offline development, tests, and low-cost fallback behavior.
StreamObserver
Observer contract for token, completion, and error events. WebSocket, SSE, logging, and tests can all implement it.
ChatService
Facade-like orchestrator for the use case. It appends user messages, builds context, invokes the provider, dispatches tools, broadcasts tokens, and stores the final answer.
Design Patterns Used
ModelProvider is the strategy. ChatService can run against OpenAI, Anthropic, or local inference without branching on provider type.
OpenAiProvider and AnthropicProvider translate vendor-specific client behavior into the common ModelProvider contract.
StreamObserver decouples token production from delivery. A CLI, browser stream, audit logger, or test spy can subscribe to the same events.
ModelProvider.create and ChatService.create centralize provider construction so callers ask for a named provider instead of knowing concrete classes.
Step-by-Step Design
1Represent the conversation as an ordered aggregate
Keep history inside Conversation and make Message immutable. Roles let the same list hold system prompts, user text, assistant output, and tool results.
public final class Message { public enum Role { SYSTEM, USER, ASSISTANT, TOOL } private final Role role; private final String content; public int estimatedTokens() { return Math.max(1, content.length() / 4); } }2Build a bounded context window before every call
A chat platform cannot send unlimited history. Walk backward through messages until the token budget is reached, while preserving message order.
public synchronized List<Message> contextWindow() { List<Message> selected = new ArrayList<>(); int used = 0; for (int i = messages.size() - 1; i >= 0; i--) { int cost = messages.get(i).estimatedTokens(); if (!selected.isEmpty() && used + cost > maxContextTokens) break; selected.add(0, messages.get(i)); used += cost; } return Collections.unmodifiableList(selected); }3Make provider choice a strategy
The service depends on ModelProvider, not on vendor SDK classes. Adapters handle prompt formatting, vendor clients, and response normalization.
public interface ModelProvider { String name(); void streamChat(List<Message> context, ToolCallHandler tools, StreamObserver observer); }4Create providers through a factory method
Central construction keeps command-line demos, tests, and dependency wiring from scattering provider-specific constructors.
public static ModelProvider create(String providerName) { switch (providerName.toLowerCase(Locale.ROOT)) { case "openai": return new OpenAiProvider(new OpenAiProvider.OpenAiClient()); case "anthropic": return new AnthropicProvider(new AnthropicProvider.AnthropicClient()); case "local": return new LocalProvider(); default: throw new IllegalArgumentException("Unknown provider"); } }5Stream tokens through observers
Providers push each token to StreamObserver.onToken. ChatService wraps observers so it can multicast events and persist the final assistant message.
6Dispatch tool calls through the service
Adapters ask ToolCallHandler.call for named functions. The service looks up registered ChatTool instances, executes them with validated arguments, records the tool result, and returns plain text to the provider.
7Keep provider adapters thin
OpenAiProvider and AnthropicProvider should translate data and stream normalized tokens, not own conversation memory, business rules, or tool registration.
Complete Java Implementation
Explanation of Every Class
Message
Immutable value object for a chat entry. Static constructors make valid roles explicit, and estimatedTokens supports context-window trimming.
Conversation
Owns ordered messages and exposes snapshots. contextWindow walks backward from the latest message to keep provider prompts under budget.
ModelProvider
Provider strategy interface. The nested ToolCallHandler is the safe callback into application tools, and create is the factory method for named providers.
OpenAiProvider
Adapter for an OpenAI-shaped client. It converts messages into a prompt, optionally requests the weather tool, and normalizes streamed chunks as observer tokens.
AnthropicProvider
Adapter for an Anthropic-shaped client. It has different prompt composition and delta streaming, but still satisfies ModelProvider.
LocalProvider
Lightweight provider strategy for development and tests. It uses the same streaming and tool-calling contract without any vendor client.
StreamObserver
Observer interface for token streaming, completion, and failure. Transport adapters implement this outside the core chat model.
ChatService
Use-case orchestrator. It appends user messages, obtains bounded context, invokes the provider, dispatches tools, multicasts stream events, and stores the final assistant message.
Dry Run
Sample input
Conversation has a system prompt and two old turns. Context limit is 60 estimated tokens. User asks: What is the weather in Bengaluru and summarize it? Provider is OpenAI; registered tool weather returns 24C, light rain.
| Step | Input/Event | Context window | Provider/tool action | Streamed output | Stored messages |
|---|---|---|---|---|---|
| 1 | sendUserMessage | System + recent turns + new user message | OpenAiProvider receives bounded context | No tokens yet | User message appended |
| 2 | Provider detects weather intent | Same bounded context | ChatService.call(weather, city=Bengaluru) | No tokens yet | Tool message stored |
| 3 | Tool result returned | Prompt now includes tool result | Provider resumes generation | OpenAI adapter streams | Assistant draft not stored yet |
| 4 | Provider completes | Unchanged | PersistingObserver builds final message | onComplete(finalMessage) | Assistant message appended |
The important separation is visible in step 2: the adapter can ask for a function, but only the service can dispatch the named tool and record the tool result.
Complexity Analysis
| Operation | Time | Space | Note |
|---|---|---|---|
| sendUserMessage | O(M + T) | O(M) | M messages are scanned only until the context budget is filled; T streamed tokens are forwarded. |
| contextWindow | O(M) | O(K) | Worst case scans all messages; K selected messages are copied into the bounded prompt. |
| tool dispatch | O(1) average | O(A) | Hash-map lookup by tool name plus a defensive copy of A arguments. |
| observer broadcast | O(O × T) | O(1) | Each of T tokens is sent to O registered observers plus the request observer. |
In real systems provider latency dominates CPU work, but interviewers still expect you to call out context-window cost and observer fan-out. A production memory policy can replace the linear scan with summarized checkpoints or token-indexed history.
Extensibility
New model provider
Implement ModelProvider in a new adapter, translate provider-specific streaming into StreamObserver, and add one branch to the factory method or dependency injection wiring.
New tool
Register another ChatTool under a safe name. Provider adapters can call it without importing tool classes.
Different memory policy
Replace Conversation.contextWindow with a tokenizer-aware policy, summarization, vector recall, or pinned system messages without changing providers.
Multiple transports
Add StreamObserver implementations for WebSocket, server-sent events, mobile push, audit logging, or tests.
Provider failover
Wrap ModelProvider with a retrying or fallback provider that delegates to OpenAI first, then Anthropic or local inference.
Alternative Designs
Provider-specific services
Create OpenAiChatService, AnthropicChatService, and LocalChatService, each owning its own flow.
Tradeoffs
Initially simple, but conversation memory, tool dispatch, and streaming persistence get duplicated and drift across providers.
Event bus for streaming
Publish token events to an internal event bus instead of calling observers directly.
Tradeoffs
Useful for distributed fan-out and replay, but overkill for the LLD core and harder to reason about in an interview.
Command objects for tools
Represent every function call as a Command with validation, authorization, and execution phases.
Tradeoffs
Great when tools become complex workflows, but a named ChatTool map is clearer for the base design.
Common Mistakes
- ×
Letting ChatService build OpenAI and Anthropic JSON directly, which destroys provider abstraction.
- ×
Waiting for the full provider response before notifying clients, so the design is not truly streaming.
- ×
Forgetting to persist the final assistant message after streaming, causing conversation memory to lose answers.
- ×
Sending the full conversation forever instead of enforcing a context window.
- ×
Allowing providers to execute arbitrary methods instead of using a named tool registry.
- ×
Mixing UI transport concerns such as WebSocket sessions into provider adapters.
- ×
Treating OpenAI, Anthropic, and local models as subclasses of Conversation instead of interchangeable strategies.
Follow-up Interview Questions
QHow would you support retries without duplicating streamed tokens?
Track a provider attempt id and only persist the final assistant message for the successful attempt. For clients, emit retry metadata or restart the stream with a clear boundary.
QHow would you add retrieval-augmented generation?
Insert a memory or retrieval step before ModelProvider.streamChat that adds retrieved snippets as system or tool messages. Keep providers unchanged.
QHow do you prevent unsafe tool execution?
Use an allow-list registry, validate arguments per tool, apply authorization before execution, add timeouts, and record tool results as messages for auditability.
QHow would you stream to both browser and logs?
Register two StreamObserver implementations. The service multicasts token, completion, and error events to both.
QHow would you handle provider-specific features like image input?
Add capability metadata or a richer request object while preserving the ModelProvider boundary. Unsupported providers can reject with a typed error.
Production Considerations
Durable storage
Persist conversations, messages, tool calls, provider attempt metadata, and final assistant responses in a repository or event log.
Backpressure and cancellation
Observers should support cancellation and slow-client handling so one browser connection cannot block provider streaming.
Provider credentials and quotas
Keep API keys outside adapters, route through a credential provider, enforce per-user quotas, and expose cost metrics by provider.
Safety and moderation
Add moderation, tool authorization, prompt-injection checks, and audit logs around user input and tool output.
Observability
Measure first-token latency, total latency, streamed token count, context size, tool-call count, errors, and provider fallback rate.
What Interviewers Look For
Did the candidate keep provider APIs behind ModelProvider and concrete adapters?
Can they explain why streaming is an Observer use case rather than a return string?
Did they bound conversation memory before provider calls?
Is tool dispatch safe, named, and owned by the service rather than by provider code?
Can they add a new provider or transport by adding classes, not rewriting the flow?
Quiz
0/5 answered
1.Why should **ChatService** depend on **ModelProvider** instead of **OpenAiProvider** directly?
2.What does **StreamObserver** primarily decouple?
3.Where should OpenAI-specific request formatting live?
4.Why does the design record tool results as messages?
5.What problem does **Conversation.contextWindow** solve?
Practice Variants
Add summarizing memory
IntermediateWhen history exceeds the token budget, summarize older turns into a pinned system memory message instead of dropping them entirely.
Add provider fallback
AdvancedCreate a provider wrapper that retries on transient errors and falls back from OpenAI to Anthropic or local inference.
Add typed tool schemas
AdvancedExtend ChatTool with a JSON-schema-like argument contract, validation errors, and per-tool authorization.
Flashcards
Cheat Sheet
Core model: Conversation owns ordered Message objects; roles are SYSTEM, USER, ASSISTANT, and TOOL.
Provider seam: ChatService depends on ModelProvider. OpenAiProvider, AnthropicProvider, and LocalProvider are interchangeable strategies and vendor adapters.
Streaming seam: Providers push tokens to StreamObserver. ChatService wraps observers to multicast tokens and persist the final assistant message.
Memory: Before every provider call, build a bounded context window. Production systems can replace the simple token estimate with a tokenizer, summarizer, or retrieval step.
Tools: Providers ask ToolCallHandler.call(name, args). ChatService dispatches to registered ChatTool instances and records tool output as a message.
Patterns: Strategy for providers, Adapter for vendor clients, Observer for streaming, Factory Method for provider creation.
References
- BookDesigning Data-Intensive Applications — Martin Kleppmann
- DocsRefactoring Guru — Strategy Pattern
- DocsOpenAI Function Calling Guide
- DocsAnthropic Messages API Documentation