Transformer architecture✎

The Transformer is a neural network architecture designed for processing sequential data, first described by Vaswani et al. in the 2017 paper "Attention Is All You Need" (NIPS). It replaced the recurrent and convolutional backbones that had previously dominated sequence modeling, relying instead entirely on attention mechanisms and position-wise feedforward layers. The key payoffs are the ability to capture long-range dependencies across an entire input at once and the ability to perform computation in parallel rather than serially — two properties that RNNs fundamentally lacked. These qualities enabled training on unprecedentedly large datasets using GPUs, catalyzing the modern era of large language models and generative AI.

Encoder–decoder structure

The original Transformer has an encoder-decoder design, with both halves consisting of a stack of N=6 identical layers. Each encoder layer has two sub-layers: a multi-head self-attention layer followed by a position-wise feedforward network. The decoder is a matching stack of 6 layers that adds a third sub-layer performing cross-attention (also called encoder-decoder attention) over the encoder's output — decoder queries attend over all encoder key-value pairs, allowing the decoder to focus on relevant parts of the input at every generation step. The decoder's own self-attention is masked: future positions are set to −∞ before the softmax, preserving the autoregressive property so that predictions for position i can only depend on outputs at positions less than i. Residual connections wrap every sub-layer, and layer normalization follows each one, so the output of each sub-layer is LayerNorm(x + Sublayer(x)). All sub-layers and embedding layers produce outputs of dimension d_model = 512.

Most modern LLMs drop one half of this structure. The field recognizes three broad families: decoder-only autoregressive models (GPT series, Llama, Claude, Gemini, Mistral) are best suited for open-ended text generation; encoder-only models like BERT produce rich bidirectional contextual representations suited for classification and named entity recognition; and encoder-decoder (sequence-to-sequence) models like T5 excel at tasks that transform one sequence into another, such as translation or summarization. See Self-attention mechanism for the core computational primitive all three rely on.

Tokenization and input embeddings

The smallest unit of language in a Transformer is a token — a subword unit, not a raw character or full word. Each token is assigned an integer ID, and the model maps these IDs to learned vector embeddings of dimension d_model. In the original paper the authors share the same weight matrix between input embeddings, output embeddings, and the pre-softmax linear transformation, scaling embedding weights by √d_model. The embedding only happens at the bottom of the stack; upper layers receive the output of the layer beneath them.

Positional encoding

Because attention operates on a set of vectors without inherent order, position information must be injected explicitly. The original paper adds a sinusoidal positional encoding to each token embedding before it enters the first layer: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)). The sinusoidal choice means any fixed offset k can be expressed as a linear function of the current position, allowing the model to learn relative positions and generalize to sequence lengths unseen during training. Experiments in "Attention Is All You Need" showed learned positional embeddings produce nearly identical results (Table 3, row E), so the choice is partly pragmatic.

Position-wise feedforward networks

Each encoder and decoder layer contains a two-layer fully connected feedforward network applied identically to each position: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂. The inner dimension is d_ff = 2048 (four times d_model), while input and output dimensions remain 512. Although the same transformation is applied at every position, the weights differ across layers.

MLP block and auxiliary features

Within each Transformer block, the feedforward layer is augmented by several features that are critical during training but easy to overlook when reading only the attention equations.

GELU activation. Modern decoder-only models (including GPT-2 and its descendants) replace the ReLU in the feedforward block with a Gaussian Error Linear Unit (GELU). The first linear projection expands dimensionality four-fold — from 768 to 3,072 in GPT-2 small — allowing the model to project token representations into a higher-dimensional space where richer, more complex patterns become detectable. The second linear projection then compresses back to the original dimensionality (768), retaining the useful nonlinear transformations while returning to a manageable size.

Layer normalization is applied twice per Transformer block — once before the self-attention sublayer and once before the MLP — normalizing activations across features to stabilize training, accelerate convergence, and reduce sensitivity to initial weight values.

Dropout randomly sets a fraction of weights to zero during training, preventing co-adaptation of neurons and forcing the model to learn more robust, generalized features. During inference dropout is deactivated, effectively ensembling the trained sub-networks.

Residual connections (skip connections), first introduced in the ResNet architecture in 2015, add each sublayer's input directly to its output before normalization. In GPT-2 they appear twice per block — surrounding both the attention and MLP sublayers — so that gradients can flow directly through the network during backpropagation without vanishing, enabling reliable training of deep stacks of blocks. GPT-2 small stacks 12 such blocks; large modern models stack many more.

Training results and model variations

On WMT 2014 English→German, the big Transformer achieved 28.4 BLEU, surpassing all prior models including ensembles by more than 2 BLEU. On English→French it reached 41.8 BLEU after 3.5 days on 8 P100 GPUs — a fraction of the training cost of previous state-of-the-art models. Ablation experiments (Table 3) showed that reducing the number of attention heads, shrinking key dimension d_k, or removing dropout all hurt quality, while bigger models consistently performed better. The model also generalized to English constituency parsing without task-specific tuning, achieving 92.7 F1 in a semi-supervised setting.

Beyond NLP

The Transformer's ability to treat any sequential (or sequentialized) data through attention has made it the dominant architecture across domains. Vision Transformers (ViTs) convert images into sequences of patch embeddings — a 224×224 image becomes 256 patches of 14×14 pixels, each linearly projected and positionally encoded — then process them with standard attention, often outperforming convolutional neural networks on image segmentation and object detection. The same architecture powers many diffusion models, multimodal vision-language models, and time-series forecasting models.

Architectural efficiency innovations

The original 2017 architecture has spawned a series of targeted innovations to overcome its core bottleneck: self-attention scales as O(n²) with sequence length, making very long contexts computationally expensive and memory-intensive.

Flash Attention restructures the memory-access pattern of the attention computation — fusing operations and tiling data to keep more of it in fast SRAM — reducing GPU memory usage and speeding up training by 2–4× without changing outputs mathematically.

Rotary Position Embeddings (RoPE), adopted by Llama and many other modern models, encode positional information by rotating query and key vectors in the complex plane rather than adding a fixed offset to embeddings. This allows better extrapolation to sequence lengths not seen during training.

Linear attention variants — Linformer, Performer, and Longformer among them — reduce attention complexity to O(n), making very long documents tractable, at the cost of approximating exact attention.

Mixture of Experts (MoE) activates only a learned subset of parameters (experts) per token rather than the full weight matrix, enabling dramatically larger models at similar per-token compute cost. Mistral's architecture and several frontier models exploit this approach.

Grouped-query attention (GQA) and sliding window attention (used in Mistral) reduce the memory cost of the key-value cache and allow efficient inference over long contexts respectively.

These innovations represent active architectural research rather than settled canon; the tradeoffs between them (approximation quality, hardware fit, generalization) remain subjects of ongoing study.