When a RAG system answers confidently and wrongly, look at retrieval first
Repeated prompt revisions and a model upgrade will not fix a system that never put the right passage in front of the model in the first place.
A team asked us to look at a document assistant that was giving wrong answers to policy questions. It had been through several prompt revisions and a model upgrade. Someone had written a fairly elaborate system prompt telling the model not to make things up. The answers were still wrong, and worse, they were wrong in a calm and well-organised way that made them read as correct.
We took the failing questions and, for each one, opened the corpus and found the passage that actually contained the answer. Then we checked whether that passage had been in the context window at generation time. Most of the time it had not. The model had never seen the answer. It had been handed a handful of plausible-looking passages about adjacent topics and asked to be helpful, and it had obliged.
No prompt fixes that. No model upgrade fixes that either, though a better model will sometimes hedge more convincingly, which makes the problem harder to notice rather than smaller.
Two systems in a trench coat
A retrieval-augmented system is two components with a very thin join between them. The retriever takes a question and returns passages. The generator takes the question plus those passages and returns prose. They fail for completely unrelated reasons, they are fixed by completely unrelated work, and the only thing they share is that the user sees a single blob of text at the end.
Most teams evaluate the blob. They collect a set of questions, run them through, and have someone read the answers and mark them good or bad, or have a model do the marking. This tells you the system is wrong more often than you would like, and tells you nothing at all about what to change. Whatever that score is, if the retriever nearly always surfaces the right passage you have a generation problem, and you should be looking at chunking, context ordering and the instructions. The same score with a retriever that misses the passage as often as it finds it is a retrieval problem, and every hour spent on the prompt is wasted.
End-to-end scores tell you the system is broken. They do not tell you which half.
The pattern, stated plainly
The measurement that costs a day and saves a month
The thing you want is recall at k: given a question, does the passage that contains the answer appear in the top k results the retriever hands to the model. It is a blunt measure and it is enormously more useful than anything you can compute without it.
Getting it requires labelling, and the labelling has to be done by a human who knows the domain. Take fifty to a hundred real questions, ideally from actual logs rather than invented at a whiteboard. For each one, go into the corpus and identify the chunk or chunks that support the correct answer. Record the chunk identifier next to the question. That is your set. It is tedious, it takes a knowledgeable person the better part of a day for a hundred questions, and it is the single highest-value artefact in the whole project.
- 01Collect real questionsPull from logs, support tickets, or the sessions where someone complained. Invented questions are cleaner than reality and will flatter your system.
- 02Label the supporting passage by handA domain person opens the corpus and records which chunk holds the answer. Where two chunks are both needed, record both, and note that the question requires them jointly.
- 03Measure retrieval aloneRun the retriever, ignore the model entirely, and compute how often the labelled chunk appears in the top k. Do it at k of 5, 10 and 20. The gap between 5 and 20 tells you whether the problem is ranking or candidate generation.
- 04Measure generation on perfect inputFeed the model the labelled passage directly and ask the question. Whatever fails here is a generation problem and cannot be blamed on search.
- 05Only then change somethingOne change at a time, re-run both numbers. A change that improves the end-to-end score while lowering recall has bought you a coincidence, not an improvement.
What a low recall number usually turns out to be
When recall is poor, the cause is rarely the embedding model, which is where people look first. In the systems we have taken apart, it is more often one of a small handful of unglamorous things.
- Chunking that splits the answer A definition in one chunk and its exceptions in the next. The retriever finds the definition, the model answers from it, and the exception that mattered is sitting one chunk away. Overlap helps, but the real fix is chunking along the document's own structure rather than at a fixed character count.
- Vocabulary mismatch The corpus says one thing and users say another. Dense retrieval handles synonymy well and handles internal jargon, product codes and acronyms badly. This is the case where a hybrid of vector search and plain lexical matching is worth the extra machinery, and where a Postgres deployment with both a vector index and full-text search covers most of what a specialist store would.
- Documents that were never ingested properly Scanned PDFs that produced empty text. Tables flattened into unreadable runs of numbers. Sometimes a whole directory that failed silently on the first load and has never been checked since. Count your chunks against your source files.
- Stale content ranking above current content Superseded versions of a policy sitting alongside the live one, indistinguishable to the retriever. This is a data pipeline problem wearing an AI costume.
- Multi-hop questions treated as single-hop Questions that need two passages joined together. No amount of ranking improvement fixes these, because the retriever is being asked to do reasoning. Label them separately and accept they are a different piece of work.
The half nobody instruments
Every labelled set we have seen built by a team on its own contains only questions that have answers. This is understandable and it leaves the most damaging failure mode entirely unmeasured.
Users ask things the corpus does not cover. They ask about last quarter when the documents stop at the quarter before. They ask about a jurisdiction the handbook does not mention. They ask about something that was decided in a meeting and never written down. In all of these cases the retriever will return its top k passages anyway, because that is what retrievers do. There is no threshold below which most systems return nothing. The model then receives several passages of vaguely related material and a question, and the path of least resistance is to synthesise something that sounds like an answer.
Correct behaviour here is refusal: saying that the documents do not cover it, and ideally saying what they do cover nearby. Almost nobody tests for this. Put fifteen or twenty unanswerable questions into your labelled set, marked as having no supporting passage, and score the system on whether it declines. Write them deliberately: things just outside the corpus boundary, things the corpus discusses in passing without answering, things that were true and no longer are. In regulated and public-facing work this number matters more than the accuracy number, because a confident wrong answer about eligibility or dosage or entitlement does more damage than no answer at all.
A system that never says it does not know is not a system that always knows.
Where this approach is the wrong answer
If your corpus is small enough to fit in a context window, do not build retrieval. Put the documents in the prompt and skip the entire category of problem described above. Long contexts have made this true for far more use cases than teams assume, and we have watched people build vector infrastructure for a corpus a person could read over lunch.
If your questions are really queries over structured data, retrieval over prose is the wrong shape entirely. Questions about counts, totals and trends want SQL against a modelled warehouse, not similarity search over a text dump of a report. A surprising number of RAG projects are analytics projects that took a wrong turn early.
And if the corpus itself is contradictory, out of date, or maintained by nobody, no retrieval work will save it. The system will faithfully surface the mess. That is a content problem and it needs an owner with authority over the documents, not an engineer with an embedding model.
What to do on Monday
Take twenty questions your system currently gets wrong. For each one, find the passage that holds the answer and check whether it reached the model. That is an afternoon of work with no tooling required beyond your existing logs and the corpus. Whatever ratio comes out of that afternoon tells you which half of the system to spend the next month on, and in our experience it is not the half people expect.
The labelled set you build along the way then becomes a regression test. Every subsequent change, a new chunking strategy, a different model, a reranker, gets run against it before it ships. Without that, you are changing things and reading a few outputs, which feels like progress and is how a system ends up several prompt revisions deep with the original bug untouched.