Retrieval-augmented generation (RAG)✎

Retrieval-augmented generation (RAG) is a technique that couples the generative power of a large language model with an external information-retrieval system, allowing the model to draw on documents, databases, or web sources that were never part of its training data. Proposed in a 2020 paper (Lewis et al., Advances in NeurIPS), RAG addresses a core limitation of static LLMs: their knowledge is frozen at training time. By retrieving fresh or domain-specific content at inference time and injecting it into the prompt, RAG enables LLM-based systems to access internal company data, authoritative reference sources, or recently updated information without the cost of retraining. It also allows responses to include citations, giving users a way to verify the sources behind an answer.

RAG is closely related to the "LLM OS" framing discussed in LLM capabilities, limits, and future directions, where external retrieval acts as the long-term storage that supplements the model's limited context-window "working memory." The embeddings used in RAG vector stores are the same kind of high-dimensional semantic representations described in Word embeddings and tokenization.

Core process

A RAG pipeline has three broad phases: indexing, retrieval, and generation.

During indexing, source documents are chunked and converted into dense vector embeddings — numerical representations that encode semantic meaning — and stored in a vector database. During retrieval, an incoming user query is embedded in the same space and compared against stored vectors; the most semantically similar chunks are selected. During generation, the retrieved chunks are appended to the user's query via prompt engineering (sometimes called "prompt stuffing"), giving the LLM key information early in the prompt. The LLM then synthesizes an answer from both the retrieved context and its internal training knowledge.

Newer implementations extend this basic loop with query expansion across multiple domains, reranking of retrieved documents before generation, context selection to manage prompt length, and self-improvement mechanisms that learn from previous retrievals.

Chunking strategies

How source documents are divided into retrievable units has a significant effect on retrieval quality. Three broad approaches are used:

• Fixed-length with overlap — fast and simple; overlapping consecutive chunks preserves semantic continuity across boundaries.
• Syntax-based — sentence or paragraph segmentation using libraries such as spaCy or NLTK, respecting natural language boundaries.
• File-format-aware — respects structural units native to the content type: whole functions or classes for code files, intact <table> and <img> elements for HTML, natural page or section divisions for PDFs. Libraries such as Unstructured and LangChain support this.

Retrieval and encoding improvements

Several technical improvements have been developed at the encoding and retrieval stages:

• Sparse vs. dense vectors. Sparse vectors (dictionary-length, mostly zeros) encode word identity; dense vectors encode meaning in a compact representation. Hybrid approaches combine both to get the efficiency of sparse dot products and the semantic richness of dense embeddings.
• Late interaction (e.g., ColBERT) refines ranking by comparing individual word vectors after retrieval, improving search relevance without sacrificing retrieval speed.
• Approximate nearest neighbor (ANN) search improves retrieval efficiency over exact K-nearest-neighbor (KNN) search at scale.
• Hybrid search supplements vector retrieval with a traditional keyword search, combining both result sets before passing context to the model — useful when vector similarity misses key facts.
• Retriever optimization methods such as the Inverse Cloze Task (ICT) pre-train the retriever to learn retrieval patterns; supervised methods minimize KL divergence between retrieval probabilities and the generator's likelihood distribution to align the two components.
• RETRO (Retrieval-Enhanced Transformer) redesigns the language model architecture itself to incorporate retrieval during training, achieving competitive perplexity with a model roughly 25× smaller than purely parametric counterparts — though at the cost of an expensive training run from scratch.

Limitations and failure modes

RAG reduces but does not eliminate hallucination. As Ars Technica puts it, "It is not a direct solution because the LLM can still hallucinate around the source material in its response." Several specific failure modes deserve attention:

Context misinterpretation. An LLM may extract a statement from a retrieved source without understanding its rhetorical context — a documented case: a model retrieved a chapter title asking "Barack Hussein Obama: America's First Muslim President?" from an academic book and reported it as fact.

RAG poisoning. If the external knowledge base contains factually correct but misleading sources, or if the model must reconcile conflicting sources, it may merge outdated and updated information in a misleading way.

Uncertainty blindness. Without specific training, models may generate confident-sounding answers even when the retrieved context is insufficient to support a reliable response, rather than expressing appropriate uncertainty.

Residual retraining need. RAG reduces the frequency of model retraining but does not eliminate it; the model's internal knowledge still becomes stale, and the retrieval index must be maintained and updated.

Despite these limits, RAG remains the dominant approach for grounding LLM outputs in external knowledge, and its combination of low deployment cost, source transparency, and domain adaptability makes it a standard component of production AI systems.