RAG Explained: How Retrieval-Augmented Generation Works

Retrieval-augmented generation, RAG, is the technique of finding the documents relevant to a question and putting them into the language model’s prompt before it answers. The model’s knowledge stays fixed; what it can see changes per question. It is how a general model answers questions about your documentation, your database or last week’s meeting notes, with sources it can cite, and it is the first thing to build for any “chat with our data” project.

Why it exists#

A language model knows what was in its training data, up to a cutoff, and nothing about your private documents. Fine-tuning it on those documents is slow, expensive and poor at facts. Putting the entire document set in every prompt is impossible past a few pages. RAG resolves this: keep the documents outside the model, find the few relevant passages per question, and include only those.

The pipeline#

Indexing, done once and updated as documents change:

1. Load the documents: PDFs, web pages, database rows, transcripts.
2. Split each into chunks of a few hundred words, with some overlap.
3. Turn each chunk into an embedding: a vector of numbers where similar meanings land close together.
4. Store the vectors, with the chunk text and metadata, in an index that can find nearest neighbours quickly.

Answering, per question:

1. Embed the question with the same model.
2. Find the chunks whose vectors are nearest to it.
3. Build a prompt: instructions, the retrieved chunks, the question.
4. Send it to the language model and return the answer, with the chunk sources.

A working example#

import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")      # small, fast, runs on a CPU

# --- indexing ---
def chunk(text, size=300, overlap=50):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

documents = {"handbook.txt": open("handbook.txt").read(), "faq.txt": open("faq.txt").read()}
chunks, sources = [], []
for name, text in documents.items():
    for c in chunk(text):
        chunks.append(c)
        sources.append(name)

vectors = embedder.encode(chunks, normalize_embeddings=True)   # shape (n_chunks, 384)

# --- retrieval ---
def retrieve(question, k=4):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = vectors @ q                                         # cosine similarity
    top = np.argsort(scores)[::-1][:k]
    return [(chunks[i], sources[i], float(scores[i])) for i in top]

# --- generation ---
def answer(question, llm):
    hits = retrieve(question)
    context = "\n\n".join("[" + src + "] " + text for text, src, _ in hits)
    prompt = (
        "Answer using only the context. If the context does not contain the answer, "
        "say you do not know. Cite the source in square brackets.\n\n"
        "Context:\n" + context + "\n\nQuestion: " + question
    )
    return llm(prompt), [src for _, src, _ in hits]

Swap the numpy search for a vector database when the chunk count passes a few hundred thousand; swap the small embedder for a larger one when quality matters more than speed. The shape does not change.

Chunking#

Chunk size is the most consequential setting. Too small and a chunk lacks the context to be useful; too large and the relevant sentence is diluted by unrelated text, and fewer chunks fit in the prompt. A few hundred words with overlap of a fifth is a sensible start. Better still, split on the document’s own structure — headings, paragraphs, table rows — so chunks are self-contained units of meaning rather than arbitrary windows.

Attach metadata to every chunk: source, title, section, date, access level. It is what lets you filter (only documents this user may see, only this year) and what lets the answer cite something specific.

Embeddings#

An embedding model maps text to a vector such that texts with similar meaning have similar vectors, so “how do I reset my password” lands near a chunk that says “to change your login credentials”. That is why RAG finds relevant passages even when the words differ. Use the same model for chunks and questions, always. Changing the model means re-indexing everything.

Where it goes wrong#

Almost every bad RAG answer is a retrieval failure: the right chunk was not in the top results. Before blaming the model, log the retrieved chunks for the failing question and read them. The usual causes:

• The answer is split across chunks. Increase overlap or chunk on structure.
• Exact terms matter — a product code, an error string — and embeddings blur them. Add keyword search alongside vector search; this is hybrid retrieval and it is nearly always worth doing.
• Too many near-duplicates crowd out the one distinct chunk. Deduplicate at index time.
• The question is phrased unlike any chunk. Rewrite the question with the model first, or generate a hypothetical answer and search with that.
• Top-k is too small or too large. Retrieve more than you need, then rerank with a cross-encoder model that scores each chunk against the question properly, and keep the best few.

Evaluating#

Build a small set of questions with known correct answers and known source chunks. Measure two things separately: did retrieval return the right chunk (recall at k), and given the right chunk, did the model answer correctly. Separating them tells you which half to fix. Twenty to fifty questions, maintained as the documents change, is enough to stop guessing.

What it is not for#

RAG supplies information; it does not change how the model writes or reasons. A model that answers in the wrong tone, ignores a format, or lacks a skill needs prompting or fine-tuning. Many production systems use both: retrieval for the facts, a fine-tuned model to present them.

Where to go next#

• Fine-tuning vs RAG — when to use each.
• Large language models explained — the generation half.
• What is an MCP server? — retrieval as a tool an assistant can call.