Retrieval-Augmented Generation (RAG)

Definition

Retrieval-Augmented Generation (RAG) is a technique that, before a language model answers, searches an external document collection for passages relevant to the question, pastes those passages into the model's prompt, and asks it to answer from them — retrieve first, then generate a grounded answer. The model's weights never change; RAG is prompt construction, not training.

The reason to do this is that a large language model only knows what was in its training data, frozen at some cutoff date. It cannot cite a document it never saw, it goes stale the moment the world moves on, and when asked about something outside its training it tends to fill the gap with a fluent, confident invention — a hallucination. RAG attacks both problems at once by grounding the model in real source text fetched at question time: current facts, private company documents, and niche knowledge the model was never trained on all become answerable without retraining anything.

That grounding is also RAG's ceiling. If the search step fails to surface the passage that actually contains the answer, no amount of model capability recovers it — the model will either decline or ground its answer on whatever wrong passage it was handed. Most disappointing RAG deployments are not suffering from a weak language model. They are suffering from a retriever that returns passages topically related to the question that do not answer it.

How It Works

RAG has two phases: a one-time indexing phase done offline, and a retrieval-and-generation phase that runs on every question.

Indexing. The source documents are first split into chunks — passages small enough to retrieve precisely and to fit several into a prompt. Chunk size is a design choice, not a fixed constant; modern systems commonly use something in the range of 500–1,000 tokens per chunk. Each chunk is then passed through an embedding model, which turns it into a single dense vector — a list of numbers that positions the chunk in a semantic space where texts about similar things sit close together. The dimensionality is another convention set by the embedding model you pick: 768 is typical for a BERT-base encoder, and 1,536 is common for widely used commercial embedding APIs. Every chunk vector, with a pointer back to its text, goes into a vector database built for fast nearest-neighbor search.

Retrieval and generation. When a question arrives, it is embedded with the same model into a vector in the same space. The database then runs a vector search for the top-k nearest chunks — typically ranked by cosine similarity, with k often around 5 — and those chunk texts are pasted into the model's prompt ahead of the question. This is semantic search: it matches on meaning rather than exact keywords, which is what lets a question phrased one way retrieve a passage phrased another. (Classic keyword information retrieval such as BM25 is often blended in as well — a "hybrid" retriever — because a pure embedding search can miss exact names, codes, or rare terms.) The model then generates its answer conditioned on that retrieved context.

The cost of this convenience lands in the context window. The retrieved chunks are not free — they are pasted into the prompt and re-sent on every single call. Retrieve 5 chunks of roughly 800 tokens each and you have spent about 5 × 800 = 4,000 tokens of the context window on background material before the user's question or the model's answer is counted at all. Retrieve more, or larger, chunks and you pay more per call in both money and latency; retrieve too few and you risk leaving out the passage that held the answer. Choosing k is that trade-off.

Origin. The architecture and the name come from Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. (arXiv:2005.11401), presented at NeurIPS 2020. The original system was already recognizably today's pattern: it split a Wikipedia dump into "disjoint 100-word chunks, to make a total of 21M documents," retrieved candidates with a Dense Passage Retriever (DPR) over a FAISS index using a Hierarchical Navigable Small World approximation, and trained with k∈ retrieved passages. The specific numbers have moved — chunks are larger, embedding models better, indexes bigger — but retrieve-then-generate has not.

Real-World Applications

• Answer engines with citations. Perplexity AI and similar search assistants run a retrieval step over the live web and generate an answer that footnotes the pages it drew from — the citations are the retrieved chunks, which is why you can click through and check them.
• Enterprise question answering over private data. A support or internal-knowledge assistant indexes a company's own tickets, wikis, and product docs so employees or customers can ask questions in plain language and get answers grounded in documents the base model was never trained on and could not otherwise see.
• Coding assistants over a specific codebase. Retrieving the relevant files, symbols, and docs for the file you are editing, then generating a completion or explanation grounded in your repository rather than generic training data.
• Regulated-domain research support. Legal and clinical tools retrieve the specific statute, case, or guideline text and require the generated summary to quote or point back to it, so a human can verify the source — an application where an ungrounded confident guess is exactly the failure that must be prevented.

Challenges

The failures worth understanding in RAG are almost all failures of the retrieval step, because the generator can only be as good as the passages it is given.

Garbage retrieval produces a confident wrong answer. This is the central risk. If the top-k passages are topically related but do not contain the answer, the model does not refuse — it grounds its answer on the wrong chunk and presents the result with the same fluency as a correct one. The output looks like a cited, grounded answer and is simply wrong, which is more dangerous than an obvious hallucination because it wears the costume of evidence.

Chunk boundaries cut facts in half. Because documents are split before they are embedded, a fact that spans a boundary can be severed. A figure in one chunk and the caption explaining it in the next, a clause and its exception, a name and the pronoun that refers to it — split apart, neither chunk retrieves well and the answer built from either is incomplete. Chunking strategy, and overlap between adjacent chunks, exists mostly to fight this.

Vocabulary mismatch makes the right passage invisible. Retrieval can miss simply because the question and the answer use different words for the same thing — the user asks about "termination" and the document says "offboarding." Semantic embeddings narrow this gap but do not close it, which is why hybrid retrieval and query rewriting are common additions.

Every retrieved token is a recurring cost. As above, stuffing more context into the prompt raises quality only up to a point, then starts costing latency and money on every call and can even bury the relevant passage among distractors — "lost in the middle." More retrieval is not automatically better retrieval.

Code Example

The whole pipeline is short to sketch. This pseudocode shows the two phases — indexing once, then answering per question:

# --- Indexing phase (run once, offline) ---
chunks  = split(documents, size=800)            # ~500-1000 tokens per chunk
vectors = embed(chunks)                          # each chunk -> one dense vector
index   = VectorDB(vectors, metric="cosine")     # e.g. FAISS, pgvector, Pinecone

# --- Answering phase (runs on every question) ---
def rag_answer(question, k=5):
    q_vec   = embed(question)                    # SAME embedding model as the chunks
    hits    = index.search(q_vec, top_k=k)       # k nearest chunks by cosine similarity
    context = "\n\n".join(hit.text for hit in hits)
    prompt  = (
        "Answer using only the context below.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}"
    )
    return llm.generate(prompt)                  # model answers FROM the retrieved text

Two lines carry the whole idea. embed(question) must use the same model that embedded the chunks, or the question vector lands in a different space and the nearest-neighbor search is meaningless. And the answer is only ever as good as hits: if the right passage is not in those k chunks, it is not in the prompt, and the model cannot use what it was never given. That is why, in practice, most of the engineering effort in a RAG system goes into retrieval quality, not into the generation step.