LLM training and pretraining✎

Building a large language model is a two-stage (sometimes three-stage) process whose stages differ dramatically in cost, data type, and goal. At the end of it, the model is a parameters file — for Llama 2 70B, about 140 GB of float16 weights — that a small run script can execute locally. All the complexity and expense lives in how that file was created.

It's worth distinguishing a few terms that are sometimes used loosely: an architecture is the blueprint — the definition of layers and operations (e.g., GPT-2); a checkpoint is a specific set of learned weights trained on a particular dataset (e.g., gpt2-small); and model is an umbrella term that can mean either. This distinction matters when reasoning about transfer learning, since a checkpoint (not just an architecture) carries the knowledge acquired during pretraining.

Stage 1: Pretraining

The goal of pretraining is to compress a massive text corpus into a model's weights through next-token prediction. A typical large pretraining run ingests roughly 10 terabytes of raw text (web crawls, Wikipedia, books, code) and processes it on the order of 6,000 GPUs over ~12 days, at a cost around $2 million — performing approximately 10²⁴ floating-point operations. The Transformer architecture is trained to predict the next word given all preceding words, and its weights are updated via backpropagation and gradient descent each time it makes an error. The result is lossy compression: the 10 TB corpus is distilled into a ~140 GB parameters file (roughly 100× compression), not by verbatim memorization but by learning a latent world model — grammar, facts, reasoning patterns, and associations embedded in the weight matrices.

Pretraining is a form of self-supervised learning: the training objective (predict the next token) is derived automatically from the raw text itself, with no human annotation required. This unlocks essentially unlimited training data from the web. Two distinct self-supervised objectives are in common use. Causal language modeling (used by GPT-style, decoder-only models) predicts the next token given all preceding tokens — output depends on past and present but never future positions. Masked language modeling (used by BERT-style, encoder-only models) randomly masks tokens in a sentence and trains the model to predict the masked words from bidirectional context, producing richer representations for classification tasks at the cost of not being natively generative.

The environmental cost of pretraining is substantial and often under-appreciated. Training a very large model from scratch consumes massive compute, translating directly into carbon emissions — and this is before accounting for the additional cost of hyperparameter search runs. This is a primary reason why the community norm of releasing pretrained weights (as Hugging Face, Meta's Llama family, and others do) has significant practical and environmental value: it amortizes the cost across all downstream users.

The "Attention Is All You Need" paper's original training used the WMT 2014 datasets (4.5M English-German sentence pairs; 36M English-French sentences) with byte-pair encoding vocabularies (~37K tokens for EN-DE, 32K word-pieces for EN-FR), trained using the Adam optimizer with a custom learning rate schedule: warmup for 4,000 steps, then decay proportional to the inverse square root of step count. Three regularization techniques were applied: residual dropout (rate 0.1 on the base model), label smoothing (ε = 0.1), and checkpoint averaging.

A base model produced by pretraining alone does not behave like an assistant. Given a question, it may respond with more questions, because it is mimicking the statistical structure of web documents rather than executing a conversational role.

Stage 2: Finetuning (alignment)

To turn a base model into a helpful assistant, human annotators produce roughly 100,000 high-quality prompt–response pairs, and the model is trained on this curated dataset with the same next-token prediction objective. This process is far cheaper than pretraining — often a single day on a small GPU cluster. The model learns to shift from "document completion" behavior to responding as a cooperative, polite assistant. The result is what end users interact with: ChatGPT, Claude, Llama Chat, etc.

The rationale for fine-tuning rather than training from scratch for each new task is compelling on multiple grounds. The pretrained model already encodes a statistical understanding of language and world knowledge; fine-tuning leverages this rather than discarding it. The fine-tuning dataset can therefore be much smaller — the knowledge is "transferred," hence the term transfer learning. And time, compute, and financial costs are all dramatically lower. For the full treatment of fine-tuning techniques — including parameter-efficient methods like LoRA, adapters, and representation fine-tuning — see Fine-tuning and parameter-efficient adaptation.

Stage 3: RLHF (optional)

State-of-the-art models add a third stage: Reinforcement learning from human feedback (RLHF). Rather than writing ideal responses (which is hard and slow), annotators compare multiple model-generated answers and rank them. This comparison data trains a reward model, and the LLM is then optimized via reinforcement learning — typically proximal policy optimization (PPO) — to maximize reward model scores while staying close to the supervised fine-tuned model via a KL-divergence penalty. The analogy from narrow domains is instructive: DeepMind's AlphaGo first imitated expert human moves, then achieved superhuman performance through self-play RL where the reward was simply winning. The challenge for general LLMs is that open-ended language lacks an automatic, objective reward function — judging whether an essay is "good" requires human judgment, making robust automated reward in open domains an active research problem. Direct alignment alternatives such as DPO have emerged that bypass the reward model entirely; see Reinforcement learning from human feedback (RLHF) for the full treatment.

Scaling laws

LLM performance — measured as next-token prediction loss — follows a smooth power-law relationship with two variables: N (number of parameters) and D (amount of training data). No plateau has been observed: as compute, parameter counts, and dataset sizes grow, prediction error decreases steadily. Critically, better next-token prediction correlates empirically with better downstream performance on college exams, coding benchmarks, and professional certifications. This means that — absent algorithmic breakthroughs — companies can guarantee improved models simply by scaling hardware and data, a finding with profound commercial implications.

The reversal curse

Because pretraining processes tokens left-to-right, knowledge in the weights can be strikingly unidirectional. A model trained on "Tom Cruise's mother is Mary Lee Pfeiffer" will correctly answer "Who is Tom Cruise's mother?" but may fail "Who is Mary Lee Pfeiffer's son?" — the reverse lookup was never directly modeled. This illustrates that LLMs have not stored a relational database but learned statistical patterns over sequences, with all the asymmetries that entails.