Word embeddings and tokenization✎

Computers cannot process raw text; language must first be translated into numbers. Modern LLMs do this in two steps: tokenization breaks text into subword units, and embedding maps those units into high-dimensional vectors that carry semantic meaning. Both steps happen before any attention computation begins, and the quality of these representations shapes everything downstream in the Transformer architecture.

Tokenization

The smallest unit of language in a Transformer is a token — typically a subword, not a full word or single character. Each token is assigned a unique integer ID. Subword tokenization schemes such as byte-pair encoding (BPE) and word-piece (used in BERT) offer a practical middle ground: common words get their own token, while rare or novel words are split into recognizable pieces, dramatically reducing vocabulary size and the computational cost of processing text. The original "Attention Is All You Need" paper used BPE vocabularies of ~37,000 tokens for English–German and 32,000 word-pieces for English–French. GPT-2 and GPT-3 use a vocabulary of 50,257 tokens.

Importantly, the vocabulary is fixed before training begins. Splitting happens at a semantic seam — the word "empowers," for instance, may become two tokens — meaning token boundaries don't always align with morphological intuitions. The full vocabulary size directly determines the width of the output probability distribution the model must compute at each generation step.

Embeddings as geometric space

Each token ID is mapped to a learned real-valued vector. GPT-2 (small) represents each of its 50,257 tokens as a 768-dimensional vector, stored in an embedding matrix of shape (50,257 × 768) — approximately 39 million parameters in this layer alone. In GPT-3 these vectors have 12,288 dimensions; in the original Transformer base model, 512. The dimension is a hyperparameter of the architecture and determines the "width" of all subsequent computation in the model.

The placement of vectors in this space is not arbitrary — training causes semantically similar words to cluster together. This geometric structure supports vector arithmetic that reveals meaning: the classic example is that vector("woman") − vector("man") ≈ vector("queen") − vector("king"), meaning gender and royalty relationships are encoded as consistent directional offsets in the space.

Alignment between two vectors is measured by their dot product: multiplying corresponding components and summing. A large positive dot product indicates strong alignment or similarity — the same operation that underlies the Self-attention mechanism's scoring of query-key pairs.

From tokens to contextual representations

Raw token embeddings are context-free: the word "model" gets the same initial vector whether it appears in a sentence about machine learning or fashion. Each Transformer block updates these vectors based on surrounding context through attention, gradually shifting each token's representation toward what the full sentence implies. By the time the final layer runs, the vector for each token carries a rich, context-dependent meaning rather than a bare lexical identity.

Positional information

Token embeddings alone carry no information about order. The Transformer adds positional encoding vectors to each token embedding before the first attention layer, allowing the model to distinguish "dog bites man" from "man bites dog." See Transformer architecture for the sinusoidal encoding formula used in the original paper. Different models handle this differently: GPT-2, rather than using a fixed sinusoidal formula, trains its own positional encoding matrix from scratch alongside the rest of the model, integrating positional knowledge directly into the training process. More recent models like Llama use Rotary Position Embeddings (RoPE), which encode position through rotation of the query and key vectors in the attention computation rather than via additive offsets to the input embeddings.