Videocaptured 2026-06-29 · processed
But what is a GPT? (Transformer Architecture Explained)
# But what is a GPT? (Transformer Architecture Explained)
## Introduction
Modern generative artificial intelligence—from text-to-image generators like DALL-E 3 to conversational agents like ChatGPT—is powered by a neural network architecture known as the **Transformer**. First introduced by researchers at Google in the seminal 2017 paper *"Attention Is All You Need"*, the Transformer was originally designed for machine translation (e.g., translating English to Chinese). However, its design proved highly versatile, fundamentally changing the landscape of natural language processing (NLP) and broader artificial intelligence.
---
## The Core Mechanism of Text Generation
At its heart, a Generative Pre-trained Transformer (GPT) generates text by predicting the next word in a sequence.
- **Iterative Prediction:** The model does not produce an entire essay or paragraph in a single step. Instead, it takes a prompt, calculates the probability distribution for the next likely word, selects one, appends it to the input, and repeats the process.
- **Autoregressive Nature:** Because each step relies on the previously generated words, the model's output is fed back into itself recursively to build coherent, long-form narratives.
---
## Machine Learning Models and Parameters
To understand how a Transformer processes text, it is helpful to define what a "model" is in machine learning:
- **Mathematical Functions:** A machine learning model is a complex mathematical function that maps inputs (such as pixels of an image or sequences of words) to outputs (such as a label or a predicted next word).
- **Tunable Parameters (Weights):** Unlike traditional software with rigid, hand-coded instructions, neural networks learn patterns from massive datasets. They do this by adjusting numerical values called **parameters** or **weights**.
- **Scale of Modern Models:**
- A simple linear regression model might have only two parameters (slope and y-intercept).
- Deep learning models scale this to extreme proportions. For instance, **GPT-3** contains **175 billion parameters**, represented as adjustable knobs that dictate how information flows through the network.
---
## High-Dimensional Word Embeddings
Computers cannot process raw text; they require numerical representations. This translation is achieved through **word embeddings**:
- **Words as Vectors:** In deep learning, every word in a vocabulary is represented as a high-dimensional vector (a list of real numbers). In GPT-3, these vectors live in a space with **12,288 dimensions**.
- **Semantic Space:** The placement of these vectors is not arbitrary. During training, the model positions vectors such that words with similar meanings or grammatical roles are clustered close together.
- **Vector Arithmetic:** Because words are represented as geometric coordinates, we can perform mathematical operations that reveal semantic relationships:
$$\text{Vector}(\text{"woman"}) - \text{Vector}(\text{"man"}) \approx \text{Vector}(\text{"queen"}) - \text{Vector}(\text{"king"})$$
Similarly, relationships between historical figures, countries, and concepts can be traversed geometrically.
- **The Dot Product:** To determine how closely aligned two word vectors are, the model computes their **dot product** (multiplying corresponding components and summing the results). A high positive dot product indicates strong alignment or similarity.
---
## The Transformer Block: Attention and MLPs
A Transformer consists of stacked processing units. Inside each unit are two primary components: **Attention blocks** and **Multilayer Perceptrons (MLPs)**.
### The Attention Mechanism
In isolation, a word like "model" is ambiguous—it could refer to a machine learning algorithm or a fashion runway walker. The **attention mechanism** resolves this ambiguity by allowing word vectors to "talk" to one another.
- It dynamically updates a word's vector representation based on the surrounding context.
- By calculating attention scores between words, the network shifts the vector for "model" closer to "computer science" or "fashion" depending on the nearby words.
### Multilayer Perceptrons (MLPs)
After the attention step updates the vectors based on context, MLPs process each vector individually to extract higher-level features and transition them toward the final prediction.
### Context Size Limit
Transformers are constrained by a **context size** (e.g., 2,048 tokens in original GPT-3 models). This is the maximum number of words/tokens the network can look back on at any single moment to make its next prediction.
---
## From Vectors to Words: Unembedding and Softmax
Once the final layer of the Transformer outputs a highly refined vector representing the next word's context, this vector must be translated back into a readable word.
```
[Final Vector (12,288 dims)]
│
▼
┌──────────────────────┐
│ Unembedding Matrix │ (Maps 12,288 dims to vocabulary size, e.g., 50,257)
└──────────────────────┘
│
▼
[Logits (Raw Scores)]
│
▼
┌──────────────────────┐
│ Softmax Function │ (Normalizes scores to probabilities between 0 and 1)
└──────────────────────┘
│
▼
[Probability Distribution] -> (e.g., "boy": 96%, "child": 3%, "young": 1%)
```
### The Unembedding Matrix ($W_U$)
The model multiplies the final vector by an **unembedding matrix**. This projects the high-dimensional vector back into a space corresponding to the size of the model's vocabulary (e.g., 50,257 words/tokens). The resulting raw scores are called **logits**.
### The Softmax Function
Logits can be any real numbers (positive or negative) and do not sum to 1. To convert them into usable probabilities, they are passed through the **softmax function**:
1. Every logit $x_i$ is exponentiated ($e^{x_i}$), ensuring all values become positive.
2. Each exponentiated value is divided by the sum of all exponentiated values in the vector.
3. This outputs a mathematically valid probability distribution where every value is between $0$ and $1$, and the entire distribution sums exactly to $1$.
---
## The Role of Temperature in Generation
When generating text, we do not always want the model to choose the absolute most probable word, as this can lead to repetitive and robotic phrasing. Instead, we sample from the probability distribution, controlled by a parameter called **temperature ($T$)**.
Mathematically, temperature is introduced into the softmax formula:
$$\text{Softmax}(x_i) = \frac{e^{x_i / T}}{\sum_{j} e^{x_j / T}}$$
The value of $T$ dictates the creativity of the output:
- **Low Temperature ($T \to 0$):**
- Divides the logits by a small number, which exaggerates the differences between the highest score and the rest.
- The probability distribution becomes highly peaky (concentrated on the top choice).
- The model becomes deterministic, safe, and highly predictable.
- **Default/Moderate Temperature ($T = 1$):**
- Keeps the distribution as originally calculated during training.
- **High Temperature ($T > 1$):**
- Divides the logits by a larger number, flattening the differences between scores.
- The probability distribution becomes more uniform, giving lower-ranked, more unusual words a realistic chance of being selected.
- The model becomes highly creative, random, and prone to unexpected turns (or incoherence if set too high).