Transformer Architecture

A decoder-only transformer has three conceptual parts. First, an embedding layer maps each token id to a vector and adds positional information. Second, a stack of identical transformer blocks refines those vectors. Third, a language modeling head projects the final vectors back to a probability distribution over the vocabulary.

The important insight for interviews is uniformity. Every block has the same shape and the same interface: it takes a sequence of hidden vectors and returns a sequence of hidden vectors of the same size. This is why the architecture scales so cleanly. Making a model bigger usually means more blocks, wider hidden vectors, or more attention heads, not a new design.

Inside a block, self-attention is the only component that lets tokens exchange information. Each token produces a query, a key, and a value vector. The attention score between two tokens is the dot product of one token query and another token key, scaled and passed through softmax. Each token then builds a weighted sum of value vectors using those scores.

This is how a pronoun can attend to the noun it refers to, or how a closing bracket can attend to its opening bracket. In a decoder-only model, a causal mask prevents a token from attending to future positions, which is what makes next-token prediction well defined. The attention mechanism lesson goes deeper on the math.

After attention mixes information across positions, the feed-forward network, sometimes called the MLP, transforms each position on its own. It is usually two linear layers with a nonlinearity such as GELU in between, and it expands the hidden size by a factor of about four before projecting back.

A useful intuition is division of labor. Attention decides which other tokens matter, and the feed-forward network decides what to compute given the gathered context. A large fraction of a model parameters live in these feed-forward layers, which is why they dominate compute during the position-wise transformation.

Each sub-layer is wrapped in a residual connection: the output is the input plus the sub-layer result. This creates a residual stream that runs from the embedding all the way to the final layer, and it lets gradients flow through dozens of blocks without vanishing.

Layer normalization stabilizes the scale of activations so training stays numerically healthy. Modern models usually apply normalization before each sub-layer, a pattern called pre-norm, which trains more reliably at depth. The residual stream view is central to interpretability: each block reads from and writes to this shared stream rather than replacing it.

Stacking blocks gives the model depth. Early layers tend to capture local and syntactic structure, while later layers capture more abstract and semantic relationships. Width, meaning the hidden size and the number of attention heads, controls how much information each position can hold and how many distinct relationships attention can track at once.

For interviews, remember the levers. Parameter count grows with both depth and width. Attention cost grows with sequence length squared. Feed-forward cost grows with hidden size. These relationships explain why long context and large models are expensive, a theme that returns in the inference and context window lessons.

Self-attention compares every token to every other token, so for a sequence of length n the attention matrix has n times n entries. Doubling the context roughly quadruples attention compute and the memory for attention scores. This quadratic term is the reason long context is expensive and the reason a whole research area exists around efficient and approximate attention.

For interviews, tie this to product limits. A 128K context request does far more attention work than a 4K request, which affects both latency and price. The context window lesson expands on how the key-value cache changes this picture during generation.

Instead of one attention computation, the model splits the hidden vector into several heads, each with its own smaller query, key, and value projections. Each head can specialize, for example one head tracking subject-verb agreement while another tracks bracket matching. The head outputs are concatenated and projected back to the hidden size.

The practical tradeoff is that more heads give more distinct relationship channels but each head has a smaller dimension. Head count and head dimension are tuned together, and modern serving optimizations such as grouped-query attention reduce the key-value memory by sharing keys and values across heads.

The largest parameter groups are the token embedding matrix, the attention projection matrices, and the feed-forward layers. Feed-forward layers usually dominate because they expand the hidden size by roughly four times and then project back, at every layer. The embedding and output projection can share weights in some models to save parameters.

Knowing this helps you reason about scaling. Widening the hidden size grows feed-forward and attention weights, while adding layers multiplies the whole block. Both increase quality and cost, which is exactly the tradeoff the model selection lesson formalizes.

Encoder-only transformers such as BERT read the whole sequence bidirectionally and are ideal for producing embeddings and classifications. Decoder-only transformers such as GPT use causal masking and are ideal for generation. Encoder-decoder transformers such as the original translation model use an encoder to read the input and a decoder to generate the output.

Most chat and coding models today are decoder-only because a single causal stack is simpler to scale and serve, and it handles both understanding and generation. Embedding models used for retrieval are usually encoder-style, which is why the tokenization and embeddings lesson connects here.

Serving a transformer means holding the weights and the activations in GPU memory. Weights are fixed, but activations and the key-value cache grow with batch size and sequence length. Large models are sharded across GPUs using tensor and pipeline parallelism, and the block uniformity makes this sharding regular and predictable.

Production inference commonly uses lower precision such as FP16, BF16, or INT8 quantization to fit larger models and run faster. Normalization layers and attention softmax need care to stay numerically stable in low precision. Precision choices trade a small amount of quality for large gains in memory and throughput.

The exact architecture, including layer count, head count, normalization placement, and position encoding, must be recorded with each model version. Mismatched architecture assumptions cause silent quality regressions, so serving stacks pin model configuration alongside weights.

A rough parameter estimate helps you reason about memory before you ever load a model. This estimator counts the dominant feed-forward and attention projection weights across all layers.

The causal mask is a lower-triangular allow pattern. This snippet shows how future positions are set to a very negative value before softmax so they receive near-zero weight.