Fine-tuning and parameter-efficient adaptation
Fine-tuning is the process of adapting a neural network pretrained on one task (the upstream task) to perform a different, usually more specific task (the downstream task). It is a form of transfer learning: instead of training from scratch, fine-tuning reuses the statistical structure and world knowledge already compressed into the pretrained weights, then adjusts them on a smaller, task-specific dataset via continued gradient descent. The rationale is compelling on multiple grounds — the pretrained model already understands language, facts, and reasoning patterns, so the fine-tuning dataset can be far smaller than would be needed from scratch, and compute cost is dramatically lower. As a rule, always prefer fine-tuning a pretrained checkpoint close to the target task over training from scratch, unless a very large domain-specific corpus is available. In the LLM context, fine-tuning is the second major stage after pretraining, and it is what converts a raw base model into a helpful assistant such as ChatGPT or Llama Chat.
Additional training can be applied to all parameters of a model (full fine-tuning) or only a subset, with the remaining layers frozen (excluded from backpropagation). Full fine-tuning typically yields better results but is more expensive. Fine-tuning is usually accomplished via supervised learning on labeled examples, though it can also use weak supervision or be combined with RLHF-based objectives — as in InstructGPT and Sparrow.
Parameter-efficient fine-tuning (PEFT)
Full fine-tuning of a model with tens of billions of parameters is prohibitively expensive for most practitioners and unnecessary for many tasks. Parameter-efficient fine-tuning (PEFT) methods update only a small fraction of parameters while freezing the rest, dramatically reducing storage and compute requirements.
Adapters are lightweight modules inserted into the model's architecture — typically between transformer sublayers — that nudge the embedding space for domain adaptation. They contain far fewer parameters than the base model and are the only weights updated during fine-tuning, leaving the frozen base weights unchanged. Multiple adapters can be swapped in and out, enabling a single base model to serve many tasks.
Low-rank adaptation (LoRA) (Hu et al., 2022) is the most widely adopted PEFT technique. The key insight is that the update to a weight matrix W during fine-tuning tends to have low intrinsic rank. LoRA represents the update as the product of two small matrices — W + ΔW = W + AB, where A and B have ranks far smaller than W — and trains only A and B. A model with billions of parameters may be LoRA fine-tuned with only a few million trainable parameters, approaching the quality of full fine-tuning at a fraction of the cost. LoRA has become especially popular in the image-generation community (Stable Diffusion) and is supported natively by Hugging Face's diffusers library and its broader PEFT package for language models.
Representation fine-tuning (ReFT) (Wu et al., Stanford, 2024) takes a different approach: rather than updating weights, it learns task-specific interventions on hidden representations at inference time, while keeping the entire base model frozen. The specific method LoReFT (low-rank linear subspace ReFT) operates on representations in the linear subspace spanned by a low-rank projection matrix — the representation-based analogue of LoRA — and updates less than 1% of a model's representations. ReFT targets representations rather than parameters, distinguishing it from adapter and LoRA approaches.
Architectural considerations
For convolutional architectures, it is common to freeze early layers (which capture low-level features like edges) and fine-tune only later layers (which capture high-level, task-specific features). For Transformer-based LLMs, the choice of which layers to freeze is less settled and remains an open research problem — particularly at large scale, where even the decision of how many parameters to freeze during RLHF policy optimization affects outcomes. Open-source PEFT solutions like LoRA/adapters freeze most parameters, while large labs often fine-tune all parameters during RL phases.
A pretrained model can also be augmented with a task-specific head — a new output layer trained from scratch — while the rest of the model is fine-tuned or frozen, which is the most common pattern for adapting foundation models to classification tasks.
Robustness concerns
Fine-tuning can degrade a model's robustness to distribution shifts: a model that performs better on the target distribution after fine-tuning may perform worse on out-of-distribution inputs than the original pretrained model. One practical mitigation is to linearly interpolate the fine-tuned model's weights with those of the original pretrained model (a "model soup"). This can substantially recover out-of-distribution performance while largely preserving the in-distribution gains from fine-tuning.
Commercial fine-tuning APIs
Fine-tuning APIs are offered by several major providers, including OpenAI, Microsoft Azure OpenAI Service, and Google Cloud Platform (for PaLM models), allowing practitioners to fine-tune hosted models on proprietary data without managing training infrastructure.