Browse
Retrieval-Augmented Generation (RAG)
Grounding an LLM's output in retrieved external documents instead of relying purely on its trained-in knowledge.
What it is
Retrieval-Augmented Generation (RAG) retrieves relevant documents from an external knowledge source at query time and feeds them into the LLM's context alongside the user's question — letting the model answer using up-to-date, domain-specific, or private information it was never trained on, instead of relying solely on what it memorized during training.
The core pipeline
- Chunking: source documents are split into smaller passages (a few hundred to a couple thousand tokens each) — chunk size is a real design decision, trading context/coherence per chunk against retrieval precision.
- Embedding: each chunk is converted into a vector via an embedding model, capturing semantic meaning so similar-meaning text ends up close together in vector space, regardless of exact wording.
- Indexing: embeddings are stored in a vector database (or a vector index bolted onto an existing database) that supports fast approximate nearest-neighbor search over potentially millions of vectors.
- Retrieval: the user's query is embedded the same way, and the index returns the top-k most similar chunks — this is a similarity search, not a keyword search, so it surfaces semantically related text even without exact word overlap.
- Generation: the retrieved chunks are inserted into the LLM's prompt as context, and the model generates its answer grounded in that retrieved text rather than purely from its parameters.
Why it matters
- RAG is usually far cheaper than fine-tuning for keeping an LLM's answers current — updating the knowledge source just means re-indexing new documents, not retraining a model.
- It reduces (but doesn't eliminate) hallucination by giving the model concrete source text to draw from, and enables citing sources, since the retrieved chunks are known.
- Retrieval quality is the actual bottleneck in most RAG systems — a perfect LLM given the wrong chunks still gives a wrong answer, which is why chunking strategy, embedding model choice, and re-ranking often matter more than which LLM is used.