Design ChatGPT

The conversation orchestrator is not just a thin proxy to an LLM. It turns a user action into a controlled execution plan: validate identity, classify intent, fetch conversation state, select memory, run safety checks, choose model and tools, start streaming, persist the final assistant message, and record telemetry.

Keeping this logic outside the inference service is important. Model servers should be optimized for GPU throughput, batching, and token generation. Product orchestration changes faster because prompts, policies, routing rules, experiments, and retention requirements evolve weekly.

Users judge chat latency mostly by time to first token and smoothness of the stream, not by total completion time. A good target is 300 to 800 ms p50 first token for short prompts, under 1.5 seconds p95 for common prompts, and 20 to 80 visible tokens per second once decoding starts.

The latency budget splits into gateway and auth under 50 ms, conversation state reads under 100 ms, prompt assembly under 50 ms, moderation under 50 to 150 ms, model prefill under 200 to 900 ms depending on context size, and decode time proportional to output length. Large context requests can have multi-second first-token latency unless summaries, prefix caching, and prompt caching are used.

A chat assistant cannot naively send the entire history forever. A 128k token model can accept long context, but sending 80k tokens on every turn increases latency and cost dramatically. At $5 per 1M input tokens, a single 80k-token prompt costs $0.40 before output; at 10k QPS that becomes financially impossible.

Treat context as a hierarchy: system policy and developer instructions first, recent turns second, compact conversation summary third, retrieved long-term memories fourth, and external documents or tools only when needed. The prompt builder should have a token budget and deterministic priority rules so important instructions are never squeezed out by old chat history.

Moderation is not a single API call. Input checks block obvious abuse, self-harm escalation, malware requests, sexual content involving minors, and policy-prohibited content. Prompt-injection defenses and data-loss prevention protect system instructions, tool credentials, enterprise documents, and private user data.

Output checks are equally important because a model can drift during generation or reveal sensitive data from context. Many systems use a lightweight streaming classifier for early stop, followed by a final response classifier before persistence, logging, or sharing.

Not every prompt deserves the largest model. A production chat system often routes simple rewriting, classification, and short Q&A to a small model at $0.15 to $0.60 per 1M tokens, while using a flagship model at roughly $3 to $15 per 1M tokens for complex reasoning, coding, and ambiguous tasks.

The router can consider user tier, region, language, safety class, required latency, context length, tool needs, expected output length, current provider health, and budget. The hardest part is not the if statement; it is measuring regressions when a cheaper model silently reduces answer quality.

A chat request consumes a prefill phase over input tokens and a decode phase over generated tokens. Capacity planning should track input tokens per second, output tokens per second, active sequences, KV cache memory, and GPU utilization rather than only QPS.

For a rough interview model, assume active generations average 20 output tokens per second for premium reasoning models and 60 to 120 output tokens per second for smaller models. A million open SSE connections is a networking problem; 50k active premium generations is a GPU scheduling and cost problem.

Separate open connections from active generations. A system can have 2M open chat tabs or mobile sessions but only 50k to 150k active generations at a time. The streaming tier is sized by concurrent sockets, heartbeat bandwidth, and fanout CPU. The inference tier is sized by input tokens per second, output tokens per second, active sequences, and KV cache memory.

Example peak: 2M connected clients, 100k active generations, 700 average input tokens, 450 average output tokens, and 25 output tokens per second per active generation. That is 70M input tokens per burst and 2.5M output tokens per second while the peak lasts. A mixed fleet might route 70 percent to small models, 20 percent to general flagship models, and 10 percent to reasoning or tool-heavy paths.

For a common short prompt, target p50 first token under 800 ms and p95 under 1.5 seconds. A reasonable budget is 30 ms gateway, 40 ms auth and quota, 80 ms conversation reads, 60 ms prompt assembly, 80 ms moderation, 250 to 700 ms model prefill, and 20 to 80 ms streaming overhead.

The largest variables are input token count, model size, queueing delay, and provider cold paths. Optimizations include keeping conversations region-local, caching summaries, using prompt-prefix caching for stable system instructions, reserving GPU capacity for paid tiers, and routing small tasks to fast models that can decode 80 to 120 tokens per second.

Long context is not a license to send everything. Even with 128k tokens, the system needs a policy-aware packing algorithm. Recent user intent should usually beat old messages. Safety and system instructions should always beat retrieved memory. Tool outputs should expire quickly unless explicitly stored.

A common strategy is rolling summary plus recent turns. Every 10 to 20 turns, summarize older messages into 500 to 2,000 tokens, store the raw messages durably, and keep the last 4k to 12k tokens verbatim. Long-term memory can be retrieval-based, but it must respect user opt-in, tenancy, and deletion requirements.

During inference, the KV cache stores attention keys and values for previous tokens so decode can process one new token at a time. KV cache memory is often the limiting resource for long-context chat. A 70B-class model with long sequences can consume many GB of KV cache per hundred concurrent long conversations depending on precision, layers, and hidden size.

Continuous batching improves GPU utilization by adding and removing sequences as requests arrive and complete. The application tier should expose cancellation, max output tokens, priority, and timeout hints so the inference scheduler can avoid wasting GPU cycles on abandoned streams.

A semantic cache stores responses keyed by normalized intent, embedding vector, prompt version, policy class, model family, locale, and tenant visibility. It is best for stable, low-risk answers: grammar fixes, public explanations, standard code snippets, and repeated enterprise FAQ responses.

Avoid caching personalized, private, rapidly changing, or safety-sensitive answers. Use TTLs from minutes to days, similarity thresholds around 0.90 to 0.95, and a final guardrail check before serving cached text. Even a 10 percent hit rate can remove a large amount of GPU cost when traffic is huge.

The router should be deterministic enough to debug and dynamic enough to survive outages. It can maintain a ranked list of candidate models by task class, context size, safety rating, region, budget, and health. Each route records why it was chosen so quality regressions can be traced.

Fallback is not free. Different providers have different tokenizers, tool schemas, safety behavior, output style, and context limits. A robust design keeps provider adapters behind a common interface, uses model-specific prompt templates, and runs shadow evaluations before moving traffic.

Input moderation catches many bad requests before expensive inference, but output moderation needs stream-aware controls. A lightweight classifier can inspect partial text every 20 to 50 tokens and stop the stream if it crosses a high-confidence policy boundary. The final response can then be replaced with a safe completion.

Enterprise systems also need PII redaction, data boundary checks, prompt-injection detection, and tool-output sanitization. For high-risk tool calls, require structured arguments, policy validation, and sometimes human approval before execution.

Track cost per completed turn, not just monthly GPU spend. With public API-style prices, a premium request using 2,000 input tokens and 600 output tokens at $5 input and $15 output per 1M tokens costs about $0.019. A small-model route at $0.15 input and $0.60 output per 1M tokens costs about $0.00066 for the same token counts.

At 100M turns per day, the difference between those two routes is roughly $1.9M per day versus $66k per day before infrastructure, cache, support, and evaluation costs. This is why routing, context trimming, semantic caching, max-token defaults, and early cancellation are core architecture features.

Measure p50, p95, and p99 for gateway latency, first-token latency, tokens per second, total completion latency, stream disconnects, moderation latency, cache hit rate, model error rate, and cost per turn. Trace every request with conversation id, turn id, prompt version, route decision, provider, model id, token counts, and safety labels.

Use layered quotas: per IP, per user, per tenant, per model tier, per minute, and per day. During GPU scarcity, degrade free or anonymous users first, cap max output tokens, move simple requests to small models, and reserve premium capacity for paid or enterprise traffic.

Retry only before the model has produced user-visible tokens, or make retries explicit with a new turn attempt id. Once streaming begins, duplicate retries can create inconsistent answers. Fallback to another provider should preserve safety checks, prompt templates, and persistence semantics.

Version system prompts, tool schemas, safety rules, summarizers, model routes, and cache namespaces. Store versions with each response so regressions can be replayed. Roll out changes with canaries, holdbacks, offline evals, and online satisfaction metrics.

Conversation storage must support deletion, export, retention windows, enterprise no-training modes, regional residency, and audit logs. Long-term memory should be opt-in and separately deletable because users may want chat history without persistent personalization.

The client can stop a generation at any time. Propagate cancellation through streaming delivery, orchestrator, model router, and inference scheduler so GPU decode stops quickly. Enforce queue timeouts such as 2 seconds for interactive chat and max generation times such as 60 to 180 seconds by model tier.

Run offline regression suites for helpfulness, hallucination, tool correctness, safety, latency, and cost before prompt or model changes. Sample production traffic with privacy controls, compare route candidates by win rate, and keep human review queues for safety-sensitive failures.

The streaming tier should translate internal token events into a simple client protocol and propagate cancellation. This sketch avoids provider details and shows the shape of a safe server-side stream.

A prompt builder should make deterministic tradeoffs instead of concatenating everything. This simplified Python example keeps mandatory policy, then recent turns, then memories if budget remains.

A real router uses eval data and health signals, but even a simple design should make cost and capability explicit. The route decision should be logged with the response.

Semantic cache hits must be gated by policy and product metadata. Similar wording is not enough; the cached answer must be safe for the same tenant, prompt version, locale, and freshness window.