Introduction
Attention solved the information access problem. Every position can directly see every other position. No bottleneck, no vanishing gradients through time. But attention alone is not an architecture. It is a mechanism, one component in a larger system.
The transformer is that system. It takes the attention mechanism and wraps it in a specific arrangement of residual connections, layer normalization, and feed-forward networks that, together, create a stack of interchangeable processing blocks. Each block refines the representation. Stack enough blocks and the network learns to perform sophisticated reasoning, translation, code generation, and more.
This post covers the full architecture: every component, why it exists, and how the pieces fit together. We start with the transformer block itself, then cover the original encoder-decoder design from "Attention Is All You Need," and finish with the decoder-only architecture that powers modern LLMs like GPT, Claude, and LLaMA.
The Transformer Block
Every transformer, encoder or decoder, is built from the same repeating unit: the transformer block. A single block contains four operations:
- Layer normalization
- Multi-head attention
- Residual connection (skip connection)
- Feed-forward network (another residual connection wraps this too)
The block takes a sequence of vectors in and produces a sequence of vectors out, with the same shape. This means blocks are stackable. The output of one block feeds directly into the next.
The diagram shows the pre-norm variant, where layer normalization comes before each sub-layer. The original transformer paper used post-norm (normalize after the residual addition), but pre-norm is now standard because it produces more stable gradients during training and allows deeper stacking.
Residual Connections
The residual connections (the dashed lines and + nodes in the diagram above) are arguably the most important structural feature of the transformer. Without them, stacking many layers causes gradient degradation, where the signal becomes too weak for early layers to learn effectively.
A residual connection adds the input of a sub-layer to its output:
output = x + SubLayer(x)
This means the sub-layer only needs to learn the difference between what it receives and what it should produce. Learning a small correction is easier than learning the entire transformation from scratch.
The Residual Stream
There is a powerful way to think about transformers that comes from the residual connection structure. The input enters a residual stream, a running representation that flows through the entire network. Each attention layer and each feed-forward layer reads from the residual stream and writes back to it.
This "residual stream" view explains several transformer properties:
- Early layers can communicate with late layers through the stream, without passing through intermediate attention or FFN computations
- Layer deletion experiments show you can remove some middle layers with minimal performance loss, because the residual stream preserves the original representation
- Superposition: The residual stream can carry more features than its dimensionality would suggest, because different components write to and read from different subspaces
Layer Normalization
Layer normalization standardizes the activations across the feature dimension for each position independently:
LayerNorm(x) = gamma * (x - mean(x)) / sqrt(var(x) + epsilon) + beta
Where gamma and beta are learned parameters (scale and shift), and epsilon is a small constant for numerical stability (typically 1e-5).
Why is this necessary? Without normalization, the scale of activations can drift across layers. One layer might produce values in the range [0, 1], the next in [0, 100]. This makes training unstable because the optimal learning rate depends on the activation scale. Layer normalization keeps the scale consistent.
Pre-Norm vs Post-Norm
The original transformer paper placed layer norm after each sub-layer (post-norm):
output = LayerNorm(x + SubLayer(x))
Modern transformers use pre-norm, placing it before:
output = x + SubLayer(LayerNorm(x))
Pre-norm produces more stable gradients because the residual path is unobstructed. In post-norm, the gradient must flow through the normalization layer, which can amplify or suppress gradient components. With pre-norm, the gradient flows directly through the addition, and the sub-layer's contribution is normalized before it enters the residual stream.
The practical impact: pre-norm transformers train more stably at large depths and can often skip learning rate warmup. Post-norm transformers sometimes produce slightly better final performance but require careful tuning.
Some modern architectures use RMSNorm (Root Mean Square normalization) instead of full layer normalization:
RMSNorm(x) = gamma * x / sqrt(mean(x^2) + epsilon)
RMSNorm drops the mean subtraction and the beta parameter, reducing computation. It works because the mean-centering is often redundant given the learned scale parameter.
Positional Encoding
Self-attention is permutation equivariant: if you shuffle the input positions, you get the same output shuffled the same way. Attention has no inherent notion of position. Without positional encoding, the sentence "dog bites man" and "man bites dog" would produce identical attention patterns.
Sinusoidal Positional Encoding (Original)
The original transformer added sinusoidal functions of different frequencies to the input embeddings:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Each dimension uses a different frequency. Low-frequency dimensions encode coarse position (beginning vs. end of sequence), while high-frequency dimensions encode fine position (adjacent tokens).
This encoding has a useful property: the encoding of position pos + k can be expressed as a linear transformation of the encoding at position pos, for any fixed offset k. This means the model can learn to attend to relative positions using linear operations.
Learned Positional Embeddings
An alternative: learn a separate embedding vector for each position, just like token embeddings. Position 0 gets a learned vector, position 1 gets another, and so on. Simple and effective for fixed-length contexts.
The limitation is that learned embeddings cannot generalize beyond the maximum position seen during training. If you train with 2048 positions, position 2049 has no embedding.
Rotary Position Embedding (RoPE)
Modern LLMs predominantly use Rotary Position Embedding (RoPE), which encodes position by rotating the query and key vectors in pairs of dimensions:
RoPE(x, pos) = x * cos(pos * theta) + rotate(x) * sin(pos * theta)
Where theta varies by dimension pair, creating rotations at different frequencies (similar to sinusoidal encoding). The key insight: the dot product between two RoPE-encoded vectors depends only on their relative position, not their absolute positions.
RoPE advantages:
- Relative position awareness: Attention scores naturally capture distance between tokens
- Extrapolation: With techniques like NTK-aware scaling or YaRN, RoPE can extend to sequence lengths beyond training
- No additional parameters: The rotation is computed from position, not learned
- Decays with distance: The expected dot product between distant positions naturally decreases, providing a useful inductive bias
The Feed-Forward Network
Each transformer block contains a position-wise feed-forward network (FFN) that processes each position independently (no cross-position interaction, that is attention's job). The original design:
FFN(x) = activation(x . W_1 + b_1) . W_2 + b_2
The critical detail: the inner dimension d_ff is larger than d_model. Typically d_ff = 4 * d_model. For a model with d_model = 4096, the FFN expands to 16384 dimensions and then contracts back to 4096.
Why expand and contract? The expansion creates a high-dimensional space where the network can represent complex nonlinear functions. The contraction projects the result back to the model dimension. This is where the transformer stores factual knowledge. Research has shown that individual neurons in the FFN activate for specific concepts, patterns, or facts.
Gated FFN Variants
Modern transformers use gated variants of the FFN, most commonly SwiGLU:
SwiGLU(x) = (x . W_1 * SiLU(x . W_gate)) . W_2
Where SiLU(x) = x * sigmoid(x). The gating mechanism (x . W_gate) provides the network with a multiplicative interaction that improves expressiveness. The cost: three weight matrices instead of two, so d_ff is typically reduced to 8/3 * d_model to keep the parameter count comparable.
The Encoder-Decoder Transformer
The original "Attention Is All You Need" (Vaswani et al., 2017) introduced a transformer with two stacks: an encoder and a decoder, connected by cross-attention.
The Encoder Stack
The encoder processes the input sequence with bidirectional self-attention. Every position can attend to every other position, forward and backward. This is ideal for tasks where you have the entire input available at once (translation, classification, summarization of a given text).
Each encoder layer is a standard transformer block: layer norm, multi-head self-attention, residual connection, layer norm, FFN, residual connection.
The Decoder Stack
The decoder generates the output sequence one token at a time. It has two attention mechanisms per layer:
-
Masked self-attention: The decoder can only attend to previous positions in the output sequence. Position 5 can see positions 0-4 but not positions 6 onward. This prevents the model from "cheating" by looking at future tokens during training.
-
Cross-attention: After self-attention, the decoder attends to the encoder's output. Queries come from the decoder, but keys and values come from the encoder. This is how the decoder accesses the input sequence.
Cross-attention is the bridge between encoder and decoder. The encoder processes the full input once, producing a set of key-value pairs. The decoder then queries these key-value pairs at every layer and every generation step. This means the encoder's computation is amortized: it runs once, but the decoder references its output repeatedly.
The Decoder-Only Transformer
The encoder-decoder design works well for sequence-to-sequence tasks where the input and output are distinct (translation, summarization). But for language modeling, where the task is simply to predict the next token, the encoder is unnecessary. The decoder alone suffices.
This is the decoder-only transformer, the architecture behind GPT, Claude, LLaMA, and most modern LLMs. Remove the encoder. Remove cross-attention. Keep only the decoder stack with its masked self-attention and feed-forward layers.
The input and output use the same vocabulary, the same embedding space, and the same sequence. The model reads a sequence of tokens and predicts the next token at each position.
Causal Masking
The defining feature of the decoder-only transformer is the causal attention mask. When predicting the token at position t, the model can only attend to positions 0, 1, ..., t. It cannot see the future.
This is enforced by adding a mask to the attention scores before softmax:
scores = Q . K^T / sqrt(d_k)
scores = scores + mask # mask is 0 for allowed, -inf for blocked
attention = softmax(scores) . V
The mask is a lower-triangular matrix: positions above the diagonal are set to negative infinity, which softmax converts to zero probability.
During training, causal masking allows the model to compute the loss for every position simultaneously. Token at position 0 predicts position 1, position 1 predicts position 2, and so on. This is much more efficient than generating one token at a time, because a single forward pass produces N-1 training signals from a sequence of N tokens.
The Decoder-Only Architecture
The full decoder-only transformer:
The simplicity is the point. One block type, repeated N times. No encoder, no cross-attention. The same architecture handles any text task: translation (include the source text in the prompt), summarization (include the document in the prompt), question answering (include the context in the prompt). The task is specified through the input, not the architecture.
KV Cache: Making Inference Efficient
During autoregressive generation, the model generates one token at a time. Each new token requires a full forward pass. But there is a massive redundancy: when generating token t+1, the keys and values for positions 0 through t are exactly the same as they were when generating token t.
The KV cache stores these previously computed keys and values, so they do not need to be recomputed:
# Without KV cache (wasteful):
# For each new token, recompute K and V for ALL positions
# With KV cache (efficient):
# Step 1: Compute K, V for token 0. Store in cache.
# Step 2: Compute K, V for token 1 only. Append to cache.
# Attention uses full cache [K0, K1] and [V0, V1].
# Step 3: Compute K, V for token 2 only. Append to cache.
# Attention uses full cache [K0, K1, K2] and [V0, V1, V2].
# ...
Without the KV cache, every step recomputes the keys and values for all previous positions, so generating token t costs O(t^2) and the full sequence costs O(n^3). With the KV cache, each step computes attention for just the new token against the cached keys and values: O(n) per step, O(n^2) total.
The trade-off: the KV cache consumes memory proportional to batch_size * num_layers * 2 * num_heads * d_head * seq_len. For a 70B parameter model with a 4096-token context, the KV cache alone can consume tens of gigabytes of GPU memory.
Grouped-Query Attention (GQA)
The KV cache memory problem led to Grouped-Query Attention (GQA), a middle ground between standard multi-head attention (MHA) and multi-query attention (MQA).
In standard MHA, each attention head has its own Q, K, and V projections. With 32 heads, you store 32 sets of keys and values in the cache.
In MQA, all heads share a single K and V. This reduces cache size by a factor of num_heads, but can hurt quality because the heads lose the ability to attend to different information.
GQA groups the heads. With 32 query heads and 8 KV groups, every 4 query heads share the same K and V:
Standard MHA: 32 Q heads, 32 KV heads (full KV cache)
GQA: 32 Q heads, 8 KV heads (1/4 the KV cache)
MQA: 32 Q heads, 1 KV head (1/32 the KV cache)
GQA reduces KV cache memory by 4x (in this example) with minimal quality loss. LLaMA 2 70B, Mistral, and many modern models use GQA.
The Training Objective: Next-Token Prediction
The decoder-only transformer is trained with a remarkably simple objective: given all previous tokens, predict the next one.
Loss = -sum over t: log P(token_t | token_0, ..., token_{t-1})
This is the cross-entropy loss between the model's predicted probability distribution over the vocabulary and the actual next token. The model produces a probability for every token in the vocabulary (typically 32K-128K tokens) at every position, and the loss penalizes assigning low probability to the correct token.
The elegance of this objective: it requires no labeled data. Any text is training data. The internet contains trillions of tokens. To predict the next word well, the model must learn grammar, facts, reasoning, coding, mathematics, and more. Next-token prediction is a universal task that forces the emergence of general intelligence.
Teacher Forcing
During training, the model sees the actual tokens at every position (not its own predictions). This is called teacher forcing. At position t, the input is always the ground truth token, regardless of what the model would have predicted. This provides stable gradients and allows the parallel computation of loss at all positions simultaneously.
During inference, there is no ground truth. The model generates token by token, feeding its own predictions back as input. This mismatch between training (teacher forcing) and inference (autoregressive) is called exposure bias, but in practice, well-trained large models handle it with negligible degradation.
Scaling Laws
One of the most consequential discoveries in modern AI is that transformer performance follows predictable scaling laws. Loss decreases as a power law with respect to three factors:
L(N, D, C) ~ a/N^alpha + b/D^beta + c
Where N is the number of parameters, D is the amount of training data, and C is the compute budget (FLOPs).
The Kaplan Scaling Laws (2020)
OpenAI's initial scaling laws paper found that loss scales smoothly with model size, dataset size, and compute. Larger models are more sample-efficient: they extract more from the same data.
Chinchilla (2022)
DeepMind's Chinchilla paper refined the scaling laws and discovered that most large models were undertrained. The compute-optimal ratio is roughly 20 tokens per parameter. A 70B model should be trained on ~1.4 trillion tokens.
This had immediate impact. Instead of training a 280B model on 300B tokens (like Gopher), train a 70B model on 1.4T tokens. Same compute, better performance. Chinchilla outperformed models 4x its size.
Modern Scaling
Current practice often "over-trains" relative to Chinchilla-optimal, training smaller models on much more data than the Chinchilla ratio suggests. The reason: inference cost scales with model size, not training data. A smaller model trained on more data is cheaper to run, even if the training itself costs more.
LLaMA 7B was trained on 1T tokens, about 143 tokens per parameter, roughly 7x the Chinchilla-optimal ratio of 20. LLaMA 2 7B on 2T tokens. The reasoning: training happens once, but inference happens billions of times.
The Complete Lineage
The transformer did not appear from nowhere. Every component has a lineage:
McCulloch-Pitts (1943) gave us the artificial neuron: weighted inputs through a threshold function. Rosenblatt (1958) added learning: the perceptron updates its own weights. Backpropagation (1986) extended learning to deep networks: multi-layer perceptrons with many hidden layers. Attention (2014) solved the long-range dependency problem: dynamic, learnable focus. The Transformer (2017) assembled the pieces: attention, residual connections, layer normalization, and feed-forward networks into a stackable block that scales predictably with compute.
Each component solves a specific problem:
| Component | Problem it solves |
|---|---|
| Weighted sum | Combining multiple signals |
| Activation function | Introducing nonlinearity |
| Backpropagation | Learning in deep networks |
| Residual connections | Training very deep networks |
| Layer normalization | Stable training dynamics |
| Attention | Long-range dependencies |
| Positional encoding | Sequence order |
| Causal masking | Autoregressive generation |
| KV cache | Efficient inference |
| GQA | Memory-efficient attention |
The transformer is not a single invention. It is the careful composition of a dozen solutions, each addressing a specific failure mode of earlier architectures. Biological neurons inspired mathematical neurons. Mathematical neurons stacked into perceptrons. Perceptrons stacked into deep networks. Deep networks gained attention. Attention gained structure. And the result handles language, code, images, audio, video, and robotics with the same architecture.
That is the transformer. Not a breakthrough, but a synthesis.
Key Takeaways
The transformer block is a residual stream with two sub-layers: multi-head attention and a feed-forward network, each preceded by layer normalization. Residual connections enable the training of very deep networks and create a continuous stream of information flow. Positional encoding injects sequence order into an otherwise position-agnostic architecture. The original encoder-decoder design uses cross-attention to bridge input and output sequences. The decoder-only simplification, with causal masking, powers every modern LLM. The KV cache makes autoregressive generation practical by avoiding redundant computation. GQA reduces cache memory with minimal quality loss. Next-token prediction, the training objective, is the universal task that drives the emergence of general capabilities. And scaling laws tell us that more compute, more data, and more parameters predictably improve performance, a finding that has shaped the entire trajectory of modern AI.