When a RAG-based assistant — the kind that answers using your organisation's documents — gives wrong answers, the instinctive reaction is always the same: "we need a more powerful model", or "let's rewrite the prompt". It is almost always the wrong move, and the most expensive one too. The bottleneck sits upstream, in a component that gets little attention: retrieval — the search that decides which documents end up in front of the model. If the document containing the answer never reaches its desk, no model — however large — can invent it.
This is the natural follow-up to the five lessons from our RAG platform, where we wrote that "quality is decided in retrieval". Here we explain why — and, above all, how to spot it before you spend.
The figure doing the rounds in slide decks
The figure "95% of generative AI projects fail" is everywhere. It comes from an MIT report (The GenAI Divide, 2025), according to which 95% of pilot projects produce no measurable impact on the bottom line, against $30–40 billion invested.
Three honest caveats before using that number: it is not a RAG-specific finding but one about GenAI pilots in general; "no measurable impact on the accounts" does not mean "the system doesn't work"; and the report's methodology has been challenged for the weakness of its sample relative to the headline.
That said, the report points to a cause that matches our experience in the field: the problem lies not in the quality of the models but in integration — in the context they are given. The thesis of this article is therefore narrower and easier to defend: in the majority of RAG projects we have seen run aground, the root cause is the search, not the model. And it is a thesis you can measure, not just recount.
The ceiling no model can break through
A RAG system is a chain with two links: a component that searches your documents for the relevant passages, and a model that writes the answer by reading those passages. All the attention (and the budget) usually goes to the second link. But a rule as banal as it is ignored applies:
The maximum quality of the answers is capped by the quality of the search: if the right passage is not found, the model can only say "I don't know" — or make something up.
There is no third option. Changing the model shifts the choice between those two; it does not solve the problem. That is why the first sensible investment is not a bigger model: it is measuring how often the search brings back the right document. If it does so 6 times out of 10, the accuracy ceiling of the whole system is 60% — and no amount of prompt tweaking will raise it.
Why the search gets it wrong
The recurring causes are few and concrete:
- Documents split badly. To be searchable, documents are divided into fragments. If the cut is mechanical, a table ends up split in half, a rule is separated from its exception, and the fragment "in that case the deadline is reduced to 15 days" no longer says which case it is talking about.
- Different words for the same thing. The user asks "can I pay an overdue bill in instalments?", while the document talks about a "repayment plan for outstanding balances". Zero words in common: keyword search will never find it.
- The fix that creates the opposite problem. The typical response is to switch to "semantic" search, which understands meaning beyond the words. True — but it is weak precisely where the other is strong: product codes, acronyms, article numbers, proper names. Keyword search finds the error code
ERR_5012at the first attempt; semantic search "drowns" it among similar concepts. - No acceptance testing. The original sin: without a set of real questions with the expected answers, every change is a gamble and every meeting a contest of opinions.
The solution: make two searches work together
The key point: keyword search and meaning-based search fail on different, complementary questions. It is not a choice between the two — you need both.
Hybrid search does exactly that: it runs the two searches in parallel and merges their rankings with a criterion that rewards consensus — a document found by both, even if not at the top, beats a document ranked first by only one. Which is the right behaviour: a passage found both by the words and by the meaning is almost certainly the one you need.
A real example, which we verified with the numbers in hand. Question: "can I pay an already overdue invoice in instalments?". The right document — the one on the "repayment plan for outstanding receivables" — was fifth in the keyword ranking and fourth in the meaning-based one: with either technique alone it would never have made it into the top three passages read by the model, and the answer would have been wrong. With the fusion of the two rankings it rose to first place — because it was the only one present in both.
| Search strategy | Position of the right document | Does the model read it? |
|---|---|---|
| Keywords only | 5th | ✗ |
| Meaning only | 4th | ✗ |
| Hybrid (fusion of the two) | 1st | ✓ |
Two things matter for decision-makers. First: this is not exotic technology — hybrid search comes built into the main enterprise search engines (Elasticsearch, OpenSearch, Azure AI Search): you switch it on, you don't invent it. Second: after the search you need a re-ranking step, because models read best what sits at the top of the context — a detail documented in the literature that on its own noticeably changes perceived quality.
It is the same philosophy behind our ACME ECMS IA & RAG Integration platform: multiple search axes over the knowledge base and re-ranking of the passages before generation. This is not theory: it is the diagram published on that page.
How to tell whether it works (before trusting it)
The acceptance test is surprisingly simple, and almost nobody runs it. You build a test set: 50–100 real questions — collected from the people who will use the system, not invented around a table — each with the document that contains the answer. Then you measure one thing only, on the search in isolation, before the model is even involved: how often does the right document appear among the top results?
That number is the ceiling of the whole project. Measured before and after every intervention — enabling hybrid search, re-ranking, better document splitting — it turns discussions into decisions: a table shows what improved things and by how much, and you stop blaming the "weak model" for faults that lie elsewhere.
The five questions to ask whoever is building it for you
- Is there a test set with our own questions and expected answers? Can we see it?
- What percentage of the time does the search bring the right document into the top results, measured on our archive?
- Does the search use both axes — keywords and meaning — or only one?
- Is there a re-ranking step before the model reads the results?
- When an answer is wrong, can you tell us whether the right document had been found? (If yes, the problem is the generation; if no, it is the search. In our experience it is almost always the latter.)
If these questions get vague answers, the project is flying blind — whatever model sits underneath.
The bottom line
RAG does not fail because the model is stupid. It fails because it is given the wrong document to read — and is then blamed for answering badly.
Before investing in a bigger model: measure the search, enable the missing axis, add re-ranking. In our experience this is the sequence of interventions with the best ratio of result to cost — but it is experience, not a theorem: a test set built on your questions is the only referee that counts.
If your document assistant has run aground at the pilot stage, we can look inside it together — starting, of course, with the measurement.
Sources.
- MIT Media Lab, Project NANDA — The GenAI Divide: State of AI in Business 2025 (July 2025); summary in Fortune, 18 August 2025. The methodology has drawn independent criticism for the weakness of its sample.
- On rank fusion (Reciprocal Rank Fusion): Cormack, Clarke, Büttcher, SIGIR 2009; documentation from Azure AI Search, Elastic and OpenSearch. The numerical example cited was recalculated and verified by the editorial team.
- On reading degradation in long contexts: Liu et al., Lost in the Middle, TACL 2024.
