What Is RAG? Retrieval-Augmented Generation Explained

RAG gives a language model access to selected external information at answer time. That improves knowledge access, but it does not guarantee truth.

TL;DR

• Retrieval-augmented generation combines retrieval, context augmentation, and generation.
• It lets an application use information outside the model’s trained parameters, including private or recently updated material.
• A RAG system is only as reliable as its knowledge base, retrieval quality, context construction, and generation controls.
• RAG is not the same as fine-tuning, memory, or an AI agent, although one system can use all four.
• Citations can make evidence inspectable, but a citation is useful only when it actually supports the claim.

What RAG means

[Retrieval-augmented generation](/glossary/retrieval-augmented-generation/) is an architecture in which an application retrieves relevant information from an external source, adds that information to the model’s input, and asks the model to generate a response using it.

The simplest mental model has three parts:

1. Retrieval: Find material relevant to the user’s question.
2. Context augmentation: Place selected material, source information, and instructions into the model’s [context](/glossary/context/).
3. Generation: Ask the language model to answer from that augmented context.

This changes where answer-time knowledge can come from. A normal model response relies heavily on patterns and knowledge encoded during training plus whatever the user supplies. A RAG response can also use an organization’s documents, product catalog, support policies, research papers, database records, or other approved sources.

The original RAG research combined a pretrained sequence-to-sequence model with retrieved documents for knowledge-intensive NLP tasks. Production systems now use the term more broadly for retrieval-backed generation pipelines, even when their exact components differ.

Why model knowledge alone is insufficient

A large language model is not a live database. Its trained knowledge has several practical limits:

• Freshness: Training cannot contain an event or policy published afterward.
• Private knowledge: Internal documents and customer-specific data are normally absent.
• Long-tail coverage: Rare details may be missing or weakly represented.
• Provenance: The model may not identify where a statement came from.
• Update cost: Changing a document is easier than retraining a model.

RAG addresses the access problem by supplying selected evidence during inference. It does not rewrite the model’s parameters. If a refund policy changes today, the team can update the indexed source and make the new version retrievable without waiting for another model-training cycle.

The core components

Knowledge base

A [knowledge base](/glossary/knowledge-base/) is the managed collection of information the system may retrieve. It can contain text documents, structured records, pages from multiple repositories, or references to source systems.

The knowledge base is a logical content layer. It is not identical to a [vector database](/glossary/vector-database/). A vector database may store searchable representations for part of the knowledge base, while canonical documents live elsewhere.

Embeddings and vector search

An [embedding](/glossary/embedding/) represents content as a numerical [vector](/glossary/vector/) designed to preserve useful relationships. The system can embed document passages and a user query, then search for vectors that are close under a similarity measure.

This is often called dense or semantic retrieval. It helps when the query and relevant passage use different words. Vector search is common, but it is not mandatory: keyword search, database filters, knowledge graphs, or [hybrid search](/glossary/hybrid-search/) can also retrieve evidence.

Retrieved context

The retriever normally returns candidates, not a finished answer. The application may filter, [rerank](/glossary/reranking/), deduplicate, trim, and organize them before constructing the model input.

Context construction matters because a model has a finite [context window](/glossary/context-window/). More retrieved text is not automatically better. Irrelevant passages consume tokens and can distract the model from stronger evidence.

Grounded generation and citations

[Grounding](/glossary/grounding/) means connecting the response to supplied evidence or other verifiable constraints. A common prompt tells the model to answer only from retrieved material, state when evidence is insufficient, and cite source identifiers.

This reduces unsupported improvisation, but it is not a proof system. A model can misunderstand a passage, merge incompatible sources, or attach a citation to a claim the source does not support. Citation quality requires claim-to-source verification, not merely displaying links.

A practical example

Imagine an employee asks:

Can I carry unused vacation days into next year?

A model relying on general knowledge may describe common HR practices but cannot know this employer’s current policy.

A RAG system can:

1. search the approved HR knowledge base;
2. retrieve the current leave-policy section;
3. exclude an archived version through metadata;
4. add the relevant passage and source date to context;
5. generate an answer that states the carryover rule;
6. link to the policy page and quote only the necessary evidence.

If the documents conflict, the correct behavior is not to invent certainty. The system should surface the conflict or route the question to HR.

RAG versus fine-tuning

RAG and fine-tuning change different things.

Dimension	RAG	Fine-tuning
Main purpose	Supply external information at inference time	Change model behavior through additional training
Updating knowledge	Update the source or index	Prepare data and train again
Provenance	Can return supporting sources	Does not inherently reveal a source
Best fit	Private, changing, or document-backed knowledge	Style, format, task behavior, or specialized patterns

Fine-tuning can help a model follow a domain format or perform a recurring task. It is usually a poor substitute for a frequently changing policy library. Many production systems use fine-tuning for behavior and RAG for evidence.

RAG versus memory

RAG retrieves knowledge relevant to the current request. [Agent memory](/glossary/agent-memory/) stores and recalls information from an agent’s past interactions, state, or learned experience.

The two can use similar infrastructure. A vector database might store both product documentation and summarized user preferences. Their meaning, ownership, retention, and update rules remain different. Read [RAG vs Agent Memory](/rag-vs-agent-memory/) for the full boundary.

RAG versus an AI agent

RAG is a knowledge-access architecture. An [AI agent](/glossary/ai-agent/) is a goal-directed system that can decide what to do, use tools, observe results, update state, and continue through an [agent loop](/how-ai-agents-work/).

A single-pass RAG assistant may retrieve once and answer. An agent may decide whether retrieval is needed, reformulate the query, search several sources, ask a follow-up question, take an action, and verify the result. RAG can be one capability inside that agent. See [RAG vs AI Agent](/rag-vs-ai-agent/) for the decision guide.

What RAG does not solve

RAG has important failure modes:

• relevant evidence may not be indexed;
• chunk boundaries can separate facts that belong together;
• retrieval may return plausible but wrong passages;
• source documents may be outdated, contradictory, or malicious;
• context may exceed the model’s usable attention;
• the model may ignore, distort, or overgeneralize evidence;
• access controls may be applied incorrectly;
• citations may not support the generated claims.

RAG can reduce some hallucinations, but “retrieved” does not mean “true.” The system needs document governance, retrieval evaluation, answer evaluation, permissions, logging, and a safe response when evidence is missing.

Builder implications

Start with the information problem, not the vector database.

Ask:

• Which questions require external evidence?
• What sources are authoritative?
• How quickly do they change?
• Who may access each source?
• What should happen when evidence conflicts?
• How will retrieval and final answers be evaluated separately?

A small, well-governed collection with strong evaluation is often more useful than a huge uncurated index. The next article, [How RAG Works](/how-rag-works/), follows the complete pipeline from document indexing to answer generation.

My Take

RAG is best understood as controlled context delivery. Retrieval is valuable only when the system can select the right evidence, preserve its authority and freshness, fit it into context, and make the generator respect its limits.

The goal is not to make the model sound knowledgeable. The goal is to make useful knowledge available at the moment of decision—and to show when it was not available.

Sources

• Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.
• Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020.
• Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023.