Why RAG exists
A model's knowledge comes from training data, which brings two hard limits: it is stale (nothing after the training cut-off) and it is generic (nothing about your internal docs). Retraining is expensive and slow. RAG takes the direct route: before answering, retrieve the relevant fragments from your material, put them in the context, and let the model answer from those fragments.
The minimal pipeline
- Chunk: split documents into 200–800 word units by meaning, never whole files;
- Embed: convert chunks to vectors with an embedding model and store them;
- Retrieve: embed the question and take the 3–8 most similar chunks;
- Rerank (optional): reorder candidates with a finer model to improve hit rate;
- Assemble: chunks + question + constraints (answer only from the chunks; say so when missing);
- Cite: label which chunk each claim came from so a human can check.
A bigger context window is not automatically better
| Approach | Upside | Cost |
|---|---|---|
| Stuff everything into a long context | Simple to build | Expensive, slow, key facts get lost |
| Retrieve top-k chunks | Cheap and focused | Depends entirely on retrieval quality |
| Retrieve + rerank | Higher hit rate | Extra call, more engineering |
Practitioners often describe a "lost in the middle" effect: in a very long context, models attend most to the beginning and the end. Placing critical chunks first or last in the prompt usually beats simply making the context longer.
Five common pitfalls
- Chunking breaks meaning: hard character cuts truncate tables and code — split on paragraphs or headings and keep a small overlap;
- Vector search only: weak at exact matches like model numbers or names — combine with keyword search (BM25);
- Stale index: documents changed but vectors were not rebuilt, so answers cite old content;
- No citations: you cannot verify sources or tell retrieval errors from generation errors;
- No permission filtering: retrieval that ignores access control leaks content users may not see.
How to evaluate
- Prepare 30–50 real questions with reference answers, including cases where "the docs do not say";
- Measure retrieval hit rate and final answer accuracy separately to locate the failure;
- Watch whether it says "I don't know" when it should — confident wrong answers are worse than refusal.
Common questions
Does RAG eliminate hallucination? It greatly reduces invention but cannot remove it: chunks can be outdated or irrelevant, and the model can misread them. Do I need a vector database? Not at first — with a few thousand chunks a plain database or in-memory search is fine; adopt a dedicated store when scale demands it. RAG or fine-tuning? RAG injects facts; fine-tuning changes style and format. They can be combined.
Try it: JSON formatter and validator — check the shape of structured retrieval output
Chunking and retrieval quality
- Chunk semantically: split on headings and paragraphs rather than mid-sentence; experiment with 200–500 characters per chunk;
- Overlap matters: 10–20% overlap between neighbours sharply reduces failures where the answer straddles a boundary;
- Filter on metadata: tag each chunk with source, date and access level, and filter before retrieval so unauthorised text is never recalled;
- Measure it: keep a set of question-to-expected-document pairs and re-run it whenever chunking changes instead of judging by feel.
Priority inside the window
With limited context, prioritise deliberately: system instructions, then the current task and constraints, then retrieved evidence, then conversation history. History is truncated first, so record conclusions in the task description rather than relying on "I said it earlier".
Filter before you retrieve
Narrow with metadata filters first, then run vector search — usually better than recalling from the whole corpus and filtering afterwards, because it improves relevance and avoids fetching content the caller may not see. Keep top-k modest: too many irrelevant fragments dilute the real evidence and lower answer quality.