RAG — retrieval-augmented generation — gives an AI agent access to your own documents at answer time, so it responds from your policies, products and data rather than from training. It works by indexing your content, retrieving the relevant passages for each question, and requiring the model to answer from them, with the source cited.
The retrieval demo always works. Someone loads a handful of clean PDFs, asks three questions, gets three good answers, and the project is approved. Then it meets the real corpus: six years of documents, four versions of the same policy, a wiki nobody has touched since a reorganisation, and two teams who disagree about the refund window.
None of that is a modelling problem, which is why it surprises people. Retrieval quality is mostly a content problem wearing an engineering costume.
What RAG actually does
- Your documents are split into passages and indexed — usually as vectors for meaning, plus keywords for exact terms.
- A question arrives, and the system searches that index for the passages most likely to contain the answer.
- Those passages are given to the model along with the question and an instruction to answer only from them.
- The answer comes back with the sources attached, so a human can verify it and the agent can say when it found nothing.
That last point is the one worth defending in design reviews. An agent permitted to answer without a source will fill the gap, and it will do so fluently. The ability to say it does not know is a feature you build in, not a limitation you tolerate.
RAG or fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts, policies, product data | Tone, format, domain vocabulary |
| Updating | Re-index the changed document | Retrain |
| Citations | Yes, by construction | No |
| Cost of a change | Minutes | A training run |
| Fails by | Retrieving the wrong passage | Confidently misremembering |
In most business systems the answer is RAG for the knowledge and careful prompting for the behaviour, with fine-tuning reserved for cases where a model needs to consistently produce a specific structure or speak a specialised dialect. Reaching for fine-tuning to fix a factual error is a costly way to make the error harder to see.
The parts that decide whether it works
Every RAG system has the same components. The difference between one that is trusted and one that is quietly abandoned comes down to a handful of decisions inside them.
- Chunking. Split by document structure — sections, headings, clauses — not by a fixed character count. A policy cut in half mid-sentence retrieves as two useless fragments.
- Hybrid search. Vectors find meaning; keyword search finds part numbers, error codes and proper nouns. Systems that use only one of the two fail in exactly the way you would predict.
- Reranking. Retrieve generously, then rerank and keep the best few. Handing a model twenty passages of which two are relevant makes the answer worse, not more thorough.
- Metadata filters. Product line, region, customer tier, document version, effective date. Most wrong answers we investigate are the right passage from the wrong context.
- Freshness. A last-verified date on every source, and content that expires when nobody has reviewed it.
- Citations. Every claim traceable to a passage, shown to the user, and available in the trace when it is not.
The unglamorous part: your content
A retrieval system inherits every flaw in the corpus it is pointed at, and then applies them at scale with a confident tone. This is the phase that takes the time and the phase teams try hardest to skip.
- Duplicates and versions. Four copies of a policy in three folders means retrieval will sometimes find the oldest.
- Contradictions. The public help centre and the internal rules routinely disagree. Someone has to decide which is authoritative, per topic.
- Undocumented knowledge. The rule that only exists in a senior agent's head cannot be retrieved. Writing those down is real project work.
- Ownership. Every source needs a person responsible for it, or freshness decays back to where it started within a quarter.
There is a consolation. Teams that do this work get value from it independently of the AI system: the contradictions found during a retrieval project were already confusing customers and new hires, silently, for years.
Permissions in retrieval
If your index contains anything not everyone should see, permissions must be enforced at retrieval time — filtered by the identity of whoever is asking, before passages reach the model. Filtering afterwards does not work: the information has already been used to compose the answer.
For customer-facing agents the safer architecture is a separate index containing only publishable content, with anything sensitive kept out entirely. Two indexes is a small cost against the class of problem it eliminates.
Evaluating retrieval, not vibes
The one artefact worth building before the system is a question set: fifty to two hundred real questions with their correct answers and the document each should come from. It takes a couple of days and it is the difference between engineering and guessing.
- Retrieval hit rate — how often the correct source appears in the retrieved set at all. If this is low, nothing downstream can save the answer.
- Answer accuracy against the known-correct response, scored by a human on a sample.
- Citation correctness — the cited source genuinely supports the claim, which is not the same as being topically related.
- Refusal correctness — when the answer is genuinely not in the corpus, does the system say so rather than improvising?
- Latency at the ninety-fifth percentile, because a support or voice agent has a budget retrieval has to fit inside.
Run that set before every change. A prompt tweak that improves three answers and breaks eleven is easy to ship and almost impossible to notice without it.
How this connects to the agent
Retrieval is not a product. It is the layer that makes everything else trustworthy: the support agent quoting a refund policy, the sales agent answering a product question mid-conversation, the voice agent checking a rule while the caller waits.
Build it once, well, and every agent that follows inherits it. Build it four times inside four projects and you will end up with four different answers to the same customer question, which is precisely the situation the project was supposed to fix.