Context Windows & KV Cache

The context window is not just the input length. It is the total number of tokens the model can condition on during a request, including system instructions, developer messages, user text, retrieved documents, tool outputs, conversation history, and generated output. If the prompt consumes the whole window, the model has no room to answer.

Typical production windows vary widely. Older and smaller models often support 4K to 8K tokens, many current general models support 16K, 32K, or 128K tokens, and specialized long-context models can reach hundreds of thousands of tokens or more. Sizes vary because model architecture, positional encoding, training data mixture, serving hardware, and product latency targets all constrain what is practical.

In prefill, the model processes the entire prompt and computes hidden states plus keys and values for all prompt tokens. Attention over the prompt compares many token pairs, so the attention work grows roughly with sequence length squared. Prefill is highly parallel but expensive for very long prompts.

In decode, the model generates one new token at a time. Each new token creates a query and attends to keys and values from earlier tokens. Without a cache, the model would repeatedly recompute past keys and values at every step. With a KV cache, decode reads the stored past and appends the new token key and value at each layer.

For each generated or prompt token, each attention layer stores two tensors: the key and the value. The rough memory intuition is 2 * layers * heads * head_dim * seq_len * batch * bytes_per_value. The leading 2 is for keys plus values. The layer count multiplies the cache because every layer has its own attention state. The sequence length and batch size multiply it because every active request needs entries for every token still in context.

In this formula, heads means the number of key-value heads held in the cache. Dense multi-head attention stores one key-value head per query head. Grouped-query attention stores fewer key-value heads shared across groups of query heads, and multi-query attention goes further by sharing one key-value head across many query heads.

Long prompts make prefill expensive because attention builds relationships across many prompt positions. Doubling prompt length can roughly quadruple the attention score work during full prompt processing. Optimized kernels reduce constants, but the underlying all-pairs pressure remains a core reason long context is slower and more costly.

Long active sequences also make decode memory expensive. The KV cache grows linearly with the number of tokens kept alive, and for large models the cache can consume more GPU memory than many engineers expect. This reduces batch size, lowers throughput, and can force requests onto larger or more GPUs.

Long context increases what the model can see, but it also increases distraction. The lost-in-the-middle effect describes models retrieving information more reliably from the beginning or end of a long prompt than from the middle. Even when the target facts are present, the model may underuse them if they are buried among less relevant tokens.

Teams also observe context rot, where accumulated history, stale decisions, repeated tool logs, and loosely related documents make the prompt less focused over time. A smaller, cleaner prompt with targeted retrieval can outperform a massive prompt full of weakly relevant material.

Prefill can use large matrix operations across the whole prompt, so GPUs handle it efficiently compared with one-token-at-a-time decode. However, full attention still needs to form relationships between prompt positions. For a prompt of length n, the attention score pattern is roughly n by n per layer and head.

This is why long-document requests can have high time-to-first-token. The model must read and process the whole prompt before producing the first output token. Faster kernels, tensor parallelism, and batching help, but they do not make a 128K prompt behave like an 8K prompt.

The cache stores keys and values, so start with 2. Multiply by layers because every transformer layer has its own attention state. Multiply by heads and head_dim because each key or value vector has per-head dimensions. Multiply by seq_len and batch because every active token in every active request needs storage. Multiply by bytes_per_value because BF16 or FP16 usually costs 2 bytes, while lower-precision cache formats cost less.

The important interview move is to say that cache memory grows linearly with context length, but the constant can be huge. A model with many layers and many key-value heads can burn gigabytes of GPU memory per long-context batch, even before considering model weights.

Classic multi-head attention gives every query head its own key head and value head. Grouped-query attention keeps many query heads but shares a smaller number of key-value heads across groups. Multi-query attention is the extreme version where many query heads share one key head and one value head.

This directly reduces the heads term in the KV cache formula. If a model has 32 query heads but only 8 key-value heads, the cache can be about one quarter of the dense key-value size. The tradeoff is that sharing can reduce attention expressiveness, so model designers tune the grouping based on quality and serving efficiency.

Long and variable-length requests create GPU memory fragmentation if each request needs a large contiguous cache allocation. PagedAttention treats the KV cache more like virtual memory: each request has logical token blocks mapped to physical GPU pages. Finished requests release pages, and growing requests receive new pages.

This improves utilization and makes high-throughput serving practical for mixed prompt lengths. It does not remove the need to store keys and values, and it does not make prefill attention quadratic cost disappear. It is a memory-management optimization that lets the GPU hold more useful active work.

Sliding-window attention limits each token to a recent window of tokens instead of the entire history. Streaming attention variants keep a bounded state or selected memory so very long streams can be processed without retaining all past tokens. These approaches reduce memory and compute but may lose information outside the retained window.

Prompt or prefix caching attacks a different problem. If requests share a prefix, such as a long system prompt or a common document, the runtime can reuse the prefix KV cache instead of recomputing it. This is most valuable when many requests share stable instructions and differ only in the final user question.

A larger context window does not guarantee better reasoning over all included tokens. Models often pay more attention to the beginning and end of a prompt than to facts buried in the middle, especially when the prompt contains many similar passages. Positioning, chunk ordering, and concise citations can matter as much as raw token budget.

Context rot is the operational version of the same problem. Long-running agents accumulate old decisions, repeated logs, obsolete plans, and irrelevant documents. The prompt becomes technically complete but semantically noisy. Good systems periodically summarize, retrieve only what is needed, and discard stale state.

Engineers often estimate GPU needs from model parameter size, then discover that long-context KV cache memory is the real limiter. Capacity plans should model maximum sequence length, average sequence length, batch size, number of layers, key-value heads, head dimension, and cache precision. Admission control should protect the system from a few huge requests starving many normal ones.

Production APIs should expose or enforce maximum input tokens, maximum output tokens, retrieval chunk limits, and per-tenant budgets. Track time-to-first-token separately from tokens-per-second because prefill and decode bottleneck differently. Long prompts usually hurt time-to-first-token, while long outputs keep decode slots and cache memory occupied.

Use retrieval to keep context small and relevant. Rank, deduplicate, compress, and cite chunks before insertion. Put critical instructions and the immediate task in stable positions, avoid dumping raw logs unless needed, and prefer concise summaries for old conversation history. The best context is the smallest context that preserves the evidence needed for the answer.

Monitor prompt tokens, output tokens, cache bytes, cache hit rate for shared prefixes, page fragmentation, eviction behavior, request rejection rate, and quality regressions by prompt length. Long-context failures often look like latency spikes, incomplete answers, or answers that ignore evidence in the middle of the prompt.

This rough estimator shows how the cache grows with model and request dimensions. Use key-value heads, not query heads, when the model uses grouped-query or multi-query attention.

This example reserves output first, then checks whether instructions, retrieval, and history fit. A production system would combine this with ranking, deduplication, and fallback summarization.