Autoregressive text generation✎

A generative LLM does not produce an entire response in a single forward pass. Instead, it generates one token at a time: it takes an input sequence, predicts the probability distribution over its entire vocabulary for the next token, samples from that distribution, appends the chosen token to the sequence, and repeats. This autoregressive loop continues until the model emits an end-of-sequence token or reaches a length limit. Every token in the output was in some sense individually chosen, making the process both flexible and sequential.

From final vector to probability distribution

After the last Transformer block processes all tokens, each position has a refined high-dimensional vector (12,288 dimensions in GPT-3, 512 in the original base model). To convert this into a prediction, the model applies an unembedding matrix (W_U) that projects the final vector into a space as wide as the vocabulary — mapping 12,288 dimensions onto ~50,257 raw scores called logits. These logits can be any real number and don't naturally sum to anything interpretable.

The softmax function converts logits into a valid probability distribution: each logit x_i is exponentiated (ensuring positivity), then divided by the sum of all exponentiated logits. The result is a vector of probabilities between 0 and 1 that sum exactly to 1. The Self-attention mechanism uses the same softmax operation internally to normalize attention scores into attention weights.

Temperature and sampling

We do not always want to pick the highest-probability token — doing so at every step produces repetitive, deterministic output. Temperature T modifies the softmax by dividing all logits before exponentiation:

Softmax(xᵢ) = e^(xᵢ/T) / Σⱼ e^(xⱼ/T)

• Low T (→ 0): Differences between logits are amplified; the distribution peaks sharply on the top token. Output is safe and predictable but can feel robotic.
• T = 1: The distribution is exactly what training produced — the baseline.
• High T (> 1): Logit differences are compressed; the distribution flattens, giving lower-ranked tokens a realistic chance. Output is more creative and surprising, but can become incoherent at extreme values.

Decoding strategies

Beyond single-token sampling, models can use more sophisticated decoding. Greedy decoding always picks the single most probable next token — fast but prone to getting stuck in repetitive loops. Beam search maintains the top-k partial sequences ("beams") simultaneously at each step, extending each, and keeping only the k most probable combined sequences; the original "Attention Is All You Need" paper used beam size 4 with length penalty α = 0.6 for its translation results.

Sampling strategies restrict the candidate pool before drawing a token, balancing diversity with coherence:

• Top-k sampling limits candidates to the k tokens with highest probability, filtering out the long tail of unlikely options. Only within this reduced set is a token sampled.
• Top-p (nucleus) sampling instead selects the smallest set of tokens whose cumulative probability mass exceeds a threshold p (e.g., p = 0.9), then samples from that set. This dynamically adjusts the candidate pool size: when the model is confident, the top few tokens may exceed the threshold; when uncertain, more candidates are included.

Temperature, top-k, and top-p can be combined. A common configuration applies temperature first (reshaping the distribution) then applies top-k or top-p filtering before the final sample. Tuning these three together lets practitioners dial between safe, predictable completions and exploratory, creative ones.

Training objective and loss

During training, the model's predicted probability distribution at each position is compared to the ground truth next token using cross-entropy loss (or equivalently, Kullback–Leibler divergence from the one-hot target). The weights are updated via backpropagation to push probability mass toward the correct token. Label smoothing (ε = 0.1 in the original Transformer) slightly redistributes probability away from the one-hot target, improving generalization even though it increases perplexity. See LLM training and pretraining for the broader training pipeline this fits into.